Monday, March 30, 2015

Biomedical funding is broken; crowdfunding is not the fix.

Last week, I wrote Calibrating crowdfunding expectations – a post based on our experience running the protocols.io Kickstarter campaign. I shared that pre-launch, we had wildly incorrect expectations about the average backing amount. We expected $70-$100, but the real mean turned out to be $39. So we had to lean on our close relatives for help, and of the $54K that we raised, $28K came from just four relatives. In this post, I want to focus on the implications of our experience for research crowdfunding.

Crowdfunding platforms can work very well for pre-sales of popular items. Even in the case of our protocols.io, while not great for funding, it was very useful to give visibility to our effort. However, while projects on Kickstarter need marketing, science projects need funding rather than advertising. Moreover, unlike ZappyLab with thousands of existing users that we asked to back our Kickstarter campaign, most scientists do not have an easy network to appeal to for crowdfunding.

Based on our experience, I was concerned that researchers turning to crowdfunding were setting themselves up for a harsh month of crowdfunding education, with very little chance of successfully raising the research dollars that they so desperately need. I know that biomedical science funding is broken, but I don’t see how crowdfunding is fixing it.

Of course, our crowdfunding was aimed at scientists to support protocols.io (a communication platform), which is very different from the research projects posted on experiment.com (the leading site for research crowdfunding) and aimed at a general audience. Therefore, to check if our experience is consistent with that of the scientists running projects on experiment.com, I took a quick look at all biology projects on the site (84 projects funded at $3K or above; spreadsheet here). The average contribution for all of them was $94, very close to our protocols.io average of $107.

The problem is that most of the 50 projects had very few backers, and that means mostly funded by close friends and relatives, which strongly skews the average towards higher numbers. I plotted the average contribution as a function of the number of backers. Figure 1 below shows that the more backers a project has, the lower the average. For the most popular projects with more than 100 backers, the averages hover around $50 – much closer to the $39 that we saw when excluding the relatives.

Figure 1. Average contribution as a function of the number of backers of the project. Numbers are from all 84  biology projects successfully funded on experiment.com at $3,000 or above.

The projects on experiment.com, just as on Kickstarter, are all-or-nothing. That is, if a project does not reach its funding goal, it gets none of the contributions. As I wrote in my previous post, there are good reasons to structure funding campaigns this way, but it also creates an extraordinary pressure on the scientist running the campaign to somehow get it to 100%. I have spoken to a number of people who ran campaigns on experiment.com, and several indicated that they had to use their own credit cards at the end to make the campaign successful. Based on this, I hypothesized that projects raising close to 100% of their total will often be rescued by relatives as was our protocols.io or will be partially self-funded by the scientist. Consistent with this prediction, graphing the average contribution as a function of the percent of the raised funding target shows a strong bias towards inflated averages around 100%-105%.

Figure 2. Average contribution as a function of the percent of funding targetNumbers are from all 84  biology projects successfully funded on experiment.com at $3,000 or above.

I know personally Cindy Wu and Denny Luan, the co-founders of experiment.com. I like and respect both of them and know that they truly have noble goals and intentions. I am publishing this post because I think transparency and correct expectations are important.

------------------
Note 1. There are also philosophical reasons to question crowdfunding for research.

Note 2. If you are planning to launch a crowdfunding campaign, take a look at the experience ofJacquelyn Gill, which is remarkably similar to our experience.

Tuesday, March 24, 2015

Calibrating crowdfunding expectations

[TL;DR summary – we raised $54,600 from 506 backers in our Kickstarter campaign. It literally saved our startup and was amazing for visibility and marketing. However, our expectations of how much we could raise were wildly off. Of the total, $28,300 came from just four relatives. The all-or-nothing model at Kickstarter misleadingly skews the “average backing” to high amounts.]

A year ago, our startup ZappyLab ran a Kickstarter campaign for the creation of protocols.io – a platform for sharing and discovering up-to-date science methods. This was the first successful crowdfunding project aimed at researchers. This Kickstarter also saved our startup. We were bankrupt, deep in the “death valley” of startup funding, with no one willing to back us. The response to our project was overwhelming; that gave us the much-needed visibility, led to our first major contract with a reagent vendor and helped to convince investors to support us.

I want to emphasize the above again – the Kickstarter saved us. I am deeply grateful to the hundreds of people who contributed, to the bloggers who wrote about it, to the countless people who spread the word. I think Kickstarter is an amazing platform and am writing this post simply to help those who plan crowdfunding campaigns. My goal is to share so that others can better calibrate their expectations and run more successful campaigns.

Before launching our project, I did a lot of research to determine the feasible funding goal. Kickstarter stats show an average pledge is $70. There is variation by category, with technology projects at an average backing of $107. Several crowdfunding gurus told us to expect a mean at or above $100 because a lot of our support would be coming from our existing users who already love and value our company and tools. We thought we could get 400-500 backers, and so we set the funding total at $50,000. We ended up raising $54.6K from 506 backers; an average of $108 per person, right around the expected range. But averages are wildly misleading. The median contribution to the protocols.io Kickstarter was $20, and the true average closer to the median than it may seem.

Within two weeks of launching our campaign, it became painfully clear that even with 500 backers, we would be nowhere near the $50K mark. Plotting our curve, we were on the path to $20,000 total. Our estimate that we could get about 500 backers was on target, but the average pledge amount was less than half of what we expected.

The tricky part is that if didn’t get to $50,000, we would get nothing and everyone watching would see a failed campaign on Kickstarter. We panicked. That didn’t help. We went to our closest relatives and begged for help. By begging for help, I don’t mean the expected leaning on friends and family to back the crowdfunded campaign. I mean an “SOS”. In the end, four relatives contributed $28,300 of the total that we raised.

Of course, our campaign is unusual for Kickstarter. The true reward we promised was better and faster science; we did not have a physical product to ship in the end. It is possible that our experience is an outlier. On the other hand, 4/6 people who ran crowdfunding campaigns told me privately that they also had the exact same experience and were forced to beg parents and other relatives to get them to the target in the end. Given our experience, I expect that this is rather common.

Kickstarter’s all-or-nothing model means that a campaign that raises $48,000 out of the target $50,000 will get zero. I understand why Kickstarter does this. This protects the backer against a campaign that raises 10% of the required total, with no product ever delivered, but the money gone. I personally would not have backed the crowdfunded projects that I did without this safety. On the other hand, this also creates an extraordinary pressure on the people running the campaign to come up with the funds by whatever means are possible. Especially for startups like ours, when investors warn that not reaching your goal is a red flag, failure is not an option.

I doubt that we are the only ones to naively launch a Kickstarter with wrong expectations. I hope this post doesn’t hurt Kickstarter, and if it helps the people running the campaigns to set better targets, this may actually result in more of the projects succeeding. To that end, more details are below.

------------------------------------------------
1. Distribution of all 506 contributions to protocols.io Kickstarter.


2. Distribution of the 502 contributions to protocols.io Kickstarter, excluding the four relatives who saved the campaign.


3. Distribution of the 486 contributions to protocols.io Kickstarter at or below $200.


4. The average was ($54,600-28,300)/502 = $52 (compared to $108 from all).

5. Of the 502 backers, 138 were friends and relatives. They contributed $12,121. That’s an average of $88. The median contribution from friends and family was $50 (compared to $20 from everyone).

6. Excluding the friends/family contributions, we got $14,014 from 361 backers, for an average of $39 – much closer to the $20 median.

7. Naturally, considering that this Kickstarter helped us survive, we are glad we did it. But if you are planning to run a crowdfunding campaign for marketing and outreach, keep in mind that it is far from trivial to pull this off. See my guide to crowdfunding here for a description of the required effort.


8. This is part 1 of a 2-part post. Next week, I will be sharing thoughts about crowdfunding for research.

Thursday, March 12, 2015

Public Service Announcement: John Wiley & Sons, you are in deep deep trouble

A week ago, Wiley published a blog post celebrating 350 years of the academic journal and patting itself on the back for the unparalleled service the publishers have been providing to the scientific community: "Why has the scholarly journal endured?" Four people weighed in with their perspectives on why the academic publishing as conceived hundreds of years ago is so wonderful today. The top two reasons were that the journal provides metrics to judge the scientists and the research published and is a good way to filter out the sea of publications and decide what to read.

The entire post is a perverse inversion of all the problems in academic publishing, presented as good reasons to keep the status quo. It is so out of touch with the conversations and concerns inside the research community, it's jarring. But the real problem for Wiley is that this is not an isolated mistake in PR; this post is representative of the culture inside Wiley and perfectly encapsulates why this corporation is about to march off the cliff.

Everyone knows that past performance is no indication of future results. Many monarchies survived way longer than 350 years, but not because of the extraordinary value they provided to their citizens. And most monarchies saw a day which marked the end of their rule.

I have been interested in science publishing since 2003 when PLOS Biology launched. I have been actively talking to publishers in the past two years. While I give the corporate publishers a hard time and am convinced that it's hard for the publisher to innovate in general, there is something unique about Wiley. Unique in a bad way.

I know many people who are or were in executive positions at all of the major publishers. I know many who were in key positions at Wiley itself. One thing that comes across in conversations with ex-Wiley executives is that this publisher is deep in denial about the looming changes. The culture inside Wiley makes it seem like a Tea Party organization, with Nature and Elsevier as the Green Party radicals. For those who know the corporate publishing world, appearing conservative on the background of the other publishers is not a good distinction.

All my conversations aside, any outside observer can tell that the publishers see the writing on the wall and are desperately trying to figure out how to stay alive and what their role will be in the future. Hence Digital Science from the Nature Publishing Group, acquisition of Mendeley by Elsevier, and the acquisitions of Papers and Biomedcentral by Springer. (UPDATE March 18: Elsevier now refers to itself as a technology company, which is a major shift. From their recent report: "[This] reflects the transformation of the company to a technology, content and analytics driven business while maintaining the link with its proud heritage.")

I would love for someone to point me to any signs of creative thinking and attempts to evolve at Wiley. Alas, I think they are busier congratulating themselves on the great service and the wonders of the impact factor. Good luck Wiley.

Sunday, January 18, 2015

Why an Accelerator Can Save You

Chris Lynch, a partner at Atlas Venture, just wrote an article titled "Accelerators claim they are in it for the long haul — I call bullshit." Because he said a lot of true things about startups and venture capital, the article is rapidly making rounds in social media with "Amen! Someone needed to say that." However, despite the many truths, Chris reached a wrong conclusion and has dispensed some flawed and possibly damaging advice by calling most accelerators "wolf in sheep’s clothing.”

I don't run an accelerator. I am not an investor. But having been in startup mode for three years as co-founder of ZappyLab, I feel compelled to respond to Chris. Particularly, I want to focus on his statement "Entrepreneurship isn’t about raising money, it’s about building businesses and creating a return on investment."

The statement itself is 100% correct; it takes many years to build a successful company. But then Chris argues that because accelerator programs are only 3-4 months long, they can't help a startup in the long term, they can only focus on the pitch and raising money, and therefore are useless. What Chris forgets is that for first-time entrepreneurs, raising money is nearly impossible. And if you go bankrupt, no matter how great your idea, you won't have those years to build the company.

Startups fail for many reason. The biggest cause of failure is a bad idea, and the second-biggest is running out of cash. It may be hard for a partner at a big venture capital firm to appreciate just how difficult it is to get that cash because the partner's daily job is giving out the money. Luckily, as a founder, I have vivid memories of how close we came to bankruptcy, despite an amazing idea, team, execution and traction.

We came up with the idea of a GitHub/wikipedia-like site for science methods in January 2012. For a month, we tried to get someone else to build it or to figure out a way to do it on the side. By April, we knew we would have to do it ourselves as a serious startup. I approached several ex-bosses and asked them if they would want to be angel investors. We got a verbal "yes" and in May, my co-founder Alexei resigned as a CTO of another startup. Raising the seed round seemed hard, but in retrospect, I realize it wasn't too bad. Within a year, we got almost $500K of seed funding from wealthy individuals that somehow were connected to us. I didn't even quit my postdoc at MIT and thought that we would quickly build everything and I would manage to succeed at both the startup and my academic career.

But Chris is right, building a company takes a long time and is hard work. By spring of 2013, it became clear that we would need about a million to build and get traction. It became clear that I would have to leave academia and focus on ZappyLab full time. My family and I moved to Berkeley to join Alexei and to be in the right place for raising the capital. I was full of confidence, thinking:

We got half a million dollars just for the idea. It was a crazy risk for the angels as there was no product, no engineers, no anything except our desire to build this. How hard can it be now to get another million considering that I joined full time, we have assembled a great team, have built an amazing prototype that scientists love, got terrific traction and media/blog coverage?

How hard? Nearly impossible. That's because we had exhausted the angels that know us personally. The next round, you are talking to institutional investors and angels who don't know you. The bar to get funding is extraordinarily high. Don't be misled by the illusion that VCs are throwing money left and right for dumb ideas. Don't compare yourself to other startups that got funded with a worse idea, product, team, etc. You don't know why they got funded. Most of them have founders who had started companies previously, and that puts them in a completely different category. Don't forget that while Silicon Valley is full of cash, there are thousands of startups competing for it, and there isn't nearly enough cash to fund them all.

Throughout 2014, we burned through our savings, maxed out our credit cards, and ran a Kickstarter to stay alive. We had to build protocols.io with virtually no resources. We had to get the users without any cash for marketing. We had to pull off miracles to survive. We did, and still, getting funding was hard.

Out of the 2.5 years of ZappyLab, it's now just the third month where we are well-funded and finally know that we won't evaporate in 6 months. Being accepted into Berkeley's Skydeck accelerator helped us a lot. Our idea has the potential to revolutionize science communication and save society billions of dollars. Vaccines, climate change, aging, cancer research - the work that impacts everyone and everything - will move significantly faster because of what we are doing. It is a great idea that might have failed had we not gotten into Berkeley's accelerator.

Yes, there are good and bad accelerators, just as there are good and bad investors in general. If you are accepted into one, take a look at Seed-DB and talk to founders that went through it. Figure out if the equity you give up is worth it. There are scams out there for sure, but they are easy to spot. There are mediocre accelerators and there are amazing ones. Do your due diligence and if it's a great accelerator, don't be greedy - the 6% you give up may make the difference between having those years to build a company or not.
//////////////////////////////////////////////

Below are some more general thoughts on accelerators.
  • As mentioned above, accelerators are different. For example, Berkeley's Skydeck does not take equity, is 6 months, and provides an amazing office space, advice, connections.
  • In addition to mentoring and help with fundraising, a key value of being in an accelerator is having the other teams around you. The network of your cohort, the advice from other founders and the support will last long beyond the 4-6 months of the program.
  • Even a small investment of $50-100K from the accelerator can do wonders for your ability to raise angel and VC cash. Angel investors and some VCs often wait for a lead investor to give them confidence. The selection by and investment from a good accelerator can serve that "lead investor" role and facilitate your fundraising long before the Demo day.




Wednesday, December 10, 2014

Exciting Science Startups

I just flew into Paris for a single day of the Axon@LeWeb event. I presented our protocols.io, in a lineup with 14 other terrific startups. But no sane person flies into Paris, in December, to spend a day in a conference hall outside of the Paris everyone knows and loves. Certainly you don’t do it for a 10-minute presentation. Of course, I may be crazy, and all startup founders are to some extent, but I had a very good reason to attend and I do not regret for a second the crazy flying and exhaustion that I am experiencing at this very moment. No regrets for two reasons:

a. I support enthusiastically Victor Henning’s effort to showcase science startups. In a sea of interesting general-audience companies at LeWeb, there is no way that an amazing company like Publons (opening up peer reviews for all to see), can get featured at the main event. Therefore, it is important to set up the precedent for an event that highlights science startups to the right audience of investors, reporters, and others in the science space. I am glad Victor did it; I hope it’s not a one-time event; and I want to help it succeed as much as possible.

b. Researchers attend science conferences for the opportunity to talk to other scientists. We don’t go to hear a specific talk. The talks are nice, but the true value of the science conference is in the productive discussion and brainstorming after the sessions. The same holds true for a meeting of scientists who founded startups. As expected, the two hours of discussion prior to our presentations, and the 6 hours of Paris metro, dinner, drinks, and more metro after the event itself – this is the part that was truly exhilarating, fun, inspiring, thought-provoking, and productive.

Throughout the course of our day together, we shared perspectives on other science startups, not at the event, but held in high regard by one or more of the assembled founders. Here are some of the science startups we like:

Rubriq: independent peer review
Riffyn software for reproducible R&D
Agave BioSystems: research and development of biological systems, sensors, diagnostics, and instrumentation.
Chematria: machine learning algorithms for medicinal discovery
Quartzy: reagent management website
Twist Bioscience:  Synthetic DNA
Biomeme: hand-held thermocycler for field studies
Authorea:  collaborative






Thursday, November 20, 2014

Startups are children; please be kind to the parents

There is a reason why you often hear entrepreneurs say something like, "I have three kids - two daughters and a startup." It's because parenting and being a founder have countless similarities. They are not identical since a startup, unlike your toddler, isn't going to bite you. And despite the biting, you don't really want to sell your child, but eventually hope to do that with the startup. Still, the comparison resonates because kids and startups are both rewarding, exhausting, exhilarating and life-changing. You never stop thinking about them, you worry ALL the time, you panic, you don't sleep - and yet, despite it all, you still have another kid and start another company.

What puzzles me is how often people lash out at founders and criticize and question our motives and decisions. Why? I have thought for a long time of writing a blog about this, and a major Twitter storm yesterday that accused us of being EVIL finally gave me the needed push.

We have given up our jobs and careers, have made enormous sacrifices, have subjected our families to financial difficulties and worries because of the crazy drive to create protocols.io. Of course I want to make money on this. But the real promise here is not the lucrative exit but the opportunity to change science communication. It's the opportunity to save society billions of wasted dollars every year and to speed up research.

So when someone accuses us of trying to stifle innovation and science communication, it hurts much more than any random mean Twitter comment ever should. It reminds me of the person at the park who criticizes a mother giving a bottle to her infant, "Don't you know that breastfeeding is so much better for your kid?" Why the assumption that the mother is a bad parent who doesn't care about the kid? Maybe the mother's milk never came in or she had a mastectomy. Maybe she tried her best and after two months of excruciating pain had a nervous breakdown and switched to formula.

Parents and founders need help and support, not vitriolic criticism. And keep in mind that neither parents nor founders are experts. At ZappyLab, we have degrees in math, molecular biology, computer science and business, but none in parenting or startups. That's because there is no PhD for these. We are all amateurs who learn on the job. That means we are all trying our best and we are all making mistakes.

If you disagree with what we are doing, just consider that there may be valid non-nefarious reasons for our decisions. But if you are sure that what we are doing is a mistake due to lack of experience, please do reach out and advise us. We crave feedback and help. Most parents and founders will welcome the advice.

Monday, November 10, 2014

Yes, papers are important for getting faculty positions

Arjun Raj has written a good post on why publishing papers is important for getting a faculty job. It is in response to my argument that having a Science/Nature/Cell (SNC) publication is not nearly as important for getting a professor job as most people think (I published a list of professors who got job offers before publishing their main postdoctoral work).

I am in violent agreement with Arjun on much of what he wrote, and disagree just as strongly with other parts of it.

Agree
"I definitely feel like my job search might have been easier with a published paper, especially in biology/medical departments. And I have definitely heard of places, for example in other countries, in which applicants have been explicitly told that the job is theirs if and only if their postdoc paper is accepted."
I never meant to suggest that publishing does not help to get a faculty job. Reading your manuscripts allows search committees to assess how you communicate your science, how you write, and how you think - all important for trying to predict whether you will be able to do good science, get funding, teach, and get others interested in your research - factors crucial for the hiring decisions, as Arjun points out. A university would be a crazy risk-taker to hire someone who has never published at all.

Disagree

"If the search committee understands the work and the researcher and believes in them both, then why does the existence of an accepted high profile paper matter so much in and of itself? A big part of the answer is that visibility matters... Having a high profile paper when you start is undeniably a part of the answer to these questions. And it’s also a simple metric of success that is readily interpreted by people across disciplines."

While publishing your work is clearly important and helpful for assessing you as a researcher, using the journal/impact factor as a proxy is the opposite of good assessment. If instead of reading your papers, the committee invites you based on the name of the journal where you published, they are using a wrong and lazy metric. Your paper in Nature may be great or terrible, and it's impossible to say which one it is, without reading the paper itself.

Faculty searches are complicated with a million factors that play a role. Hope Jahren wrote a terrific comment on how faculty searches work, describing how unique each one is and that there is no formula or method that all universities or committees use. The list of faculty who got hired prior to publishing their postdoctoral discoveries makes it clear that having a paper in a fancy journal is not a prerequisite for getting a job.

Do publications help to get a job offer? Without a doubt. Does having your paper in a glam journal help? Yes, for some search committees it increases the chances you will be invited. But the correlation between hired faculty and glam pubs is driven by the quality of their work, as my list illustrates, rather than the name of the journal being the causal factor.

As I have commented already, "What I am arguing is that it's unclear if chasing the SNC paper helps or hurts your chances of becoming faculty, all other factors being equal. That chase - the rejections, rebuttals, resubmissions, followed by more rejections, and resubmitting to another glam journal - carries a huge cost. And if that costs you advances in your research, it's not inconceivable that this very chase will hurt your chances of getting a faculty job."

I'd love to see data on whether chasing the high-impact-factor publication helps or hurts postdocs.