Ecological networks, almost-p-hacking, and conformism

Networks, of the ecological variety, are notoriously difficult to analyse. When we wrote a review on how to do it [@DelmBess18], we have often been surprised by the variety of different measures to examine what seems like essentially the same thing (looking at you, nestedness). This can make the field very difficult to navigate, not only for newcomers, but also for anyone trying to get a sense of what the best practices are. In fact, one key result from the review is that we have very little best practices that come from a thorough evaluation of the measures, how they behave, and how they relate to ecological processes.

Of course, most of ecological network research is done by loading your data into a package, and applying whatever functions are in this package to them. And there is an interesting trend, whereby the packages offers one-stop-shop solutions: call a single function, and get everything as an output. Things you need, and things you don’t. Things that have stood the test of time, and things that have only been used once. Things we know work reliably, and things we know are essentially random number generators.

And the next step is to regress all of these results on any ecological data you may have. Let’s make an analogy with statistics for a moment. The equivalent tool would be something that, given a data frame, would run all possible statistical tests, and then let the user pick whichever one lets them tell a story. Sounds like p-hacking? It is! In an ideal world, we would not even think about having functions to run everything, because we would pick, carefully, the analyses to run, and then run only these that are necessary and sufficient. We do not live in an ideal world.

None of this happens because of malice or laziness from either authors or developers. I think I can even understand the motivation: having to learn a single function seems far more user friendly than offering what would amount to dozens of functions doing a single thing. But this user-friendliness is only superficial, and the massive functions end up being extremely user-adverse! For one thing, they end up offering measures that should not be used, either at all on in the context of a specific question. It becomes much too easy to accidentally rely on measures that are not optimal, or not adapted, or just to be overwhelmed by the amount of outputs. When confronted by dozens of numbers, it may be tempting to behave like when trying to determine if the spaghetti are cooked: throw everything at the wall and see if it sticks (this is a terrible method to see if the spaghetti are cooked, and incidentally, a terrible method to decide what to measure).

But these massive functions also rob the user of any chance at adaptability. It is very rare that two studies need the exact same analysis. There are always minute details to account for, minor tweaks to perform, and steps to modify. Having a one-size-fits-all definition means that it is more work to get the analysis just like you want it. In short, these functions implicitly assume that all studies will closely follow the “textbook” example. Over the last two years, we experimented with functions that would reproduce an entire analysis (for example, “take a network, run a null model, calculate the nestedness on every random network, make a one-sample t-test”). We always ended up moving away from such solutions, because they offered no potential to fine-tune the analysis. When building our own ecological networks analysis package, EcologicalNetworks.jl, we decided to offer only “small blocks” that could be combined to produce more complex analyses. This goes somewhat against the trend in ecological networks analyses (as the reviewers noted, and this post is also a way to collect my thought about what I should answer), but I believe this is a safer way of looking at ecological networks.