Insights from Recent Prediction Markets Polls: What Users Want in a Platform

I'll Keep This Short

Welcome to my bi-weekly newsletter, “I’ll Keep This Short,” where I navigate the less-traveled paths of Artificial Intelligence (AI), building new insight beyond the banal, mainstream chatter.

I missed the last issue mid-to-late October due to some life events, but rest assured, I am still working on my series, “Fine-Tuning an Open Source Language Model,” and will get back to that in some ensuing issues.

---

I’m a casual user of a prediction markets platform called Manifold.Markets since it was launched to the wider public in early February 2022. While many of us may be aware of the concept of a financial marketplace, options, or just betting with friends on the outcome of some potential event (which I had done casually for years), I never had heard of how the idea of betting on future outcomes had been digitized as far back as even the late 1980s.

I'm interested in understanding what potential Prediction Markets users value in such a platform and what has helped make Manifold a success to date. My long-term goal is to outline what mix of design philosophies could help Manifold.Markets improve, or even help a new hypothetical platform successfully launch.

To that end, I went through a history of Digital Prediction Markets to understand where the weaknesses have been in the past, and then I did a bunch of polling of current Manifold.Markets users to understand why people find Manifold.Markets so valuable today. Without further ado, here’s what I found.

History of Digital Prediction Markets

I did a review of past prediction markets from this excellent Research Google Sheet put together by Nuño Sempere.

Some of the first internet prediction markets were the Iowa Electronic Markets, which started in 1988 and is still operational:

Another early one is the Canadian Sauder School of Business Prediction Markets, which was originally the UBC Election Stock Market, about Canadian elections and seems to have been more or less operational since 1993 after having transferred over to the Saunder School of Business around 2013, although the last update seems to have been in 2021:

Another highly active, still-extant Prediction Market is the HollyWood Stock Exchange Entertainment Market, which trades on the future potential success of movie releases.

Reviewing the Sempere Google Sheet, I created a Digital Prediction Markets timeline, with different phases, here is a rough breakdown of the history:

• Internet Prediction Markets: emerged in the late 1980s-1990s with the Iowa Electronic Markets and Hollywood Stock Exchange (and likely others). These focused on politics and entertainment. They seemed to leverage the technological paradigm of the time, networked internet more so than the web, which would not become an official protocol until 1992 and not see wider adoption until the mid to late 1990’s.

• Web Prediction Markets: In the early 2000s, now well into the web era, a wave of public play-money prediction markets launched including NewsFutures, Hubdub, and Inkling Markets. Some offered prizes. Most eventually shut down.

• Legal Gray Area Betting Markets: like Betfair and Tradesports started operating in the early to mid or late 2000s before facing legal issues in the United States, the largest country on the web during this era. Laws take time to catch up, after all.

• Cloud Era Prediction Markets: policy and public interest prediction markets like PredictIt gained traction in the early 2010’s, presumably in the advent of the cloud, along with sci-fi markets like SciCast. I artificially draw the line between early web and cloud simply because the cloud enabled so many other web apps to become more commoditized during this period in digital history.

• Blockchain Prediction Markets: launched around 2015 with Gnosis, Augur, and Hivemind but saw limited adoption, and mostly seemed to have been abandoned. Many of us are familiar with the history of the 2010’s and the growing relevance of cryptocurrency in this decade. Prediction markets appeared to have followed this hype curve.

• Regulated Markets: like Kalshi and entertainment-focused platforms like Polymarket have grown recently in the late 2010s onward as regulation seemed to start to catch up, at least for a very small niche area. Whereas Kalshi is a real-money market platform which has successfully lobbied the US Commodity Futures Trading Commission to allow itself to operate and Polymarket does not legally operate in the United States.

• Reputation Markets: Manifold.markets and Metaculus seem to be the main, “social media,” style, reputation-based prediction markets which operate like games, and while they offer in-game currency that you can buy, there is no way to redeem said currency. These markets actually seemed to have emerged as a, “pivot back to Web 2.0,” after the collapse of cryptocurrency platforms or the extremely high part of difficulty in the regulated markets platform space.

Other Parallel Elements Throughout the Timeline.

• Corporate Internal Markets: have risen and fallen over the years. Success seems to depend on continued internal buy-in. For example, Google had an internal prediction markets system which was popular for a long time. Eventually it just fell out of favor, likely because the corporate champions moved on.

• Reputation Platforms: Over the entire history, these tend to struggle with activity and accuracy compared to real-money markets. But legal issues constrain the latter. There is at least one Reputation Platform that has existed since around 1995, Foresight Exchange, which includes prediction markets from the Clinton/Dole Election of 1996. This seems to have been ported over as a student project at Carnagie Mellon in 2004, to a super durable system that just runs on an early Linux NoSQL database, and is presumably just maintained and running by one of those former students year after year indefinitely and has a low user rate (presumably by design).

The History Visualized

• First, we can look at just a high-level overview of when various digital prediction markets have been launched over time. The, “Legalism Inter-Age,” represents the time when various prediction markets appeared to have legal challenges, though this age doesn’t really appear to be over, it just came to a head for a while as the first wave of web prediction markets faced regulation challenges.

• Next we can take a look at a pie chart showing reasons for failure of the different prediction markets, if known.

  • We don’t have a lot of detailed knowledge about why every market may have shut down with about a third being un the, “unknown,” category, but the most reasonable explanation is simply that most of them go into the, “Lack of Success,” category, for some reason for another.

  • For those listed as, “Up and Running,” they may not actually have a lot of usage, so they may also be listed in the, “Lack of Success,” category depending upon how you define that.

• We can take a look at a pie chart of the duration that various prediction markets have lasted, if the shut down and start dates were known. The underlying data is a bit messy, particularly for the longer-duration platforms, in that they may technically still be in existence but might not be super active.

  • Given that, the Median Duration is 9 years and the Average Duration is 9.6 years of operation.

• To help clarify what’s going on here, let’s look into a different way of displaying the above pie chart, basically a count of how many Prediction Market Platforms exist in various categories vs. why they lasted that long. Below we show that the greatest category of prediction platform is, “still operating,” (although possibly in a zombie state without a ton of activity). The below chart also shows unknowns.

• We can also compare the duration of existence vs. the launch date to see if there was a period of time within history yet where Prediction Markets have been relatively successful in terms of how long they lasted vs. the age of history in which they launched.

  • From looking at the Duration to Date vs. Launch Year, we see that there doesn’t seem to be a clear time period when more Prediction Markets failed sooner, though the cloud and cryptocurrency recent era seemed to have more launches and short failures than the web era, which could either because we’re more aware of them due to recency bias, or because some of those earlier web-era prediction markets could have been cheaper and self-hosted, even if not super active anymore.

Further Analysis

In terms of topics, politics, sports, and entertainment see the most activity over the greatest time period. Interest in niche topics (perhaps such as the current interest in Artificial Intelligence) is often short-lived, perhaps reflecting the ebb and flow of the importance of these topics to society over time, and even hinting at Prediction Markets’ ultimate underlying mechanism as a different sort of news source.

From the comments made on the Google Sheet, public platforms need rich liquidity and engaging experiences to thrive. Most academic and corporate markets remain narrow and short-lived.

It seems that using real money betting would enhance the long-term interest in a prediction market platform as evidenced by PolyMarket’s and Kalshi’s traffic statistics, which at the time of writing sit at about 33,660 and 12,968 monthly users respectively according to SimilarWeb vs. Manifold’s self-reported roughly 10,000 monthly users. Of course PolyMarket and Kalshi have been around longer than Manifold, having started in 2020 and 2018 respectively vs. Manifold’s 2022.

From my perspective, there seems to be a huge lack of inherent trust factor barrier in any marketplace that must be overcome, and much like wildcat banking of the mid-1800’s frontier America was rife with trust and liquidity issues. Online prediction markets and the resolution of any particular marketplace on a platform is subject to the administrators, who as unregulated businesses, are extremely apt to perform rug pulls at any given time. This might help explain why while it makes sense that real money market platforms may be more popular than reputation-based systems, we currently see similar traffic at Manifold.Markets as we do at PolyMarket and Kalshi. Spending one’s money on completely speculative money just doesn’t smell right.

Tellingly, the infamous FTX run by convicted fraudster Sam Bankman Fried ran a Prediction Market on the 2020 Election.

The article I got the above image from on Coindesk, read in hindsight is even more damning regarding the potential nefarious usage of prediction markets when reading the words of SBF right on the page:

• Known for its cutting-edge cryptocurrency futures products, FTX has benefited from this non-crypto offering as well. The markets have served as a “surprisingly strong” funnel for new users, CEO Sam Bankman-Fried told CoinDesk in a private message.

• Both markets represent 22% of referral traffic to FTX, according to data from web analytics service Similar Web, the largest share of any FTX referrer, thanks to links and data posted to electionbettingodds.com.

Uh, so yeah - when perhaps the highest profile fraudster in American history since Bernie Madoff, e.g. when a literal Charles Ponzi type character is telling you, “Oh yeah, currency-backed Prediction Markets are a way we got about a quarter of the rubes in our scheme!” You should probably be wary of currency-backed prediction markets!

Side note: I should mention that both Manifold and Metaculus may have received either/or grants or funding from FTX’s Alemeda research, and I’m not quite sure how much for each one, and I’m not sure what the ramifications of that are, but it seems that Manifold may have received about a $1M equity investment.

In summary, despite my personal affinity for prediction markets, academia’s continued interest in them as a research tool, and perhaps some nice niche long-term wins, prediction markets have yet to become a widespread forecasting tool outside of certain niches. I don’t know, maybe generalized prediction markets outside of sports, election or entertainment betting are just too inherently rife with fraud and trolling to ever be something that could attract a general audience.

The outcome of elections, sports games and movie release figures are fairly public and unambiguous from a numerical objectivity standpoint, so trust might be a bit more baked-in. But what if you want to ask more difficult, deeper, and news-relevant questions?

What Is the Ideal Structure to Enable Accurate, Scalable Markets?

I have been active on Manifold.Markets lately as it seems to foster vibrant discussion while maintaining a respectful environment. The user base offers diverse perspectives - you see participation from people across the political spectrum debating issues in a largely thoughtful way. There appears to be a culture of free expression balanced with protecting other users.

However, looking at the demographics as far as I can tell the platform does seem dominated by European, American White and American Asian voices. I don't notice much participation from traditionally marginalized racial groups in the United States (or anywhere else geographically as far as I know). There also seems to be a tilt towards more affluent users.

At the same time, the active involvement of transgender users suggests Manifold has created a space welcoming of at least some identities that often face discrimination. And the tone of interactions is frequently civil, even when users disagree strongly.

Overall, Manifold Markets offers a glimpse into how to maintain ideological diversity while establishing community norms that keep dialogues productive. But there is still work to be done in making the platform inclusive of an even broader range of backgrounds. Moving forward, a focus on further minimizing toxicity and consciously including more diverse voices could make Manifold a model for constructive free speech in prediction markets.

As stated above, I'm interested in understanding what potential users value in such a platform and what has helped make Manifold a success to date, and also what could hypothetically help Manifold.Markets in the future, or help another similar platform understand the right mix of design philosophies to successfully launch. To that end, I recently conducted several polls to gain insight into user preferences, right on Manifold.Markets itself.

Key Takeaways from Polling

Do Long-Term Prediction Markets Matter?

As of the time of writing, users are split on whether decade plus long prediction markets matter. My poll on this question as of now has resulted in a nearly even split between those who think these longer-term markets are important vs. those who think they are unimportant. This indicates there is interest in both short and long-term markets. As a result, our platform should support markets with varying time horizons.

I go into this topic more deeply under, “Do Long-Term Markets Matter? Further Discussion.”

Should the Topic Space Be Curated or Non-Curated?

Currently there is a slight preference for non-curated markets. My poll asking users about curated vs. non-curated markets found a slim majority favoring non-curated. This signals that users want the freedom to create any markets that interest them, without excessive oversight. We'll need to find the right balance between open creation and some level of curation to maintain quality.

What Motivates People’s Usage of Prediction Markets?

Many users are motivated by enjoyment and gaming. When asked about reasons for using prediction markets, the largest share said they were motivated by gaming and enjoyment. While some are driven by truth-seeking and learning, fun and entertainment are big draws. This means we should design engaging, game-like experiences around predicting outcomes.

I go into this topic more deeply under, “Motivation for Usage of Prediction Markets Further Discussion.”

Should Free Thinking Take Precedence Over Preventing Harassment?

Users lean towards free thinking over preventing harassment. My poll on this issue found more users value free thinking even if it enables some objectionable speech. That said, a sizable minority care about preventing harassment and only a very tiny yet vocal minority seem to be in favor of an absolutist free-thinking approach which is essentially no-holds-bar in the, “potential to harass,” department. Hence, a properly designed Prediction Markets platform needs to take care to promote free expression while also establishing guidelines that deter harassment.

I go into more detail about this below on the section on, “Balancing Free Thinking While Preventing Harassment.”

How Does One Address the, “Alt Accounts,” Problem, If It Even Is a Problem?

The, “Alt Accounts,” problem is essentially the same as it would be on a platform such as Wikipedia - someone using alternate accounts for nefarious purposes, particularly someone using a massive number of alternate accounts. This becomes especially problematic in a reputation-based prediction markets platform, where points for usage in the game might be given away for free upon sign up.

I had two fundamental polls on this topic, first asking what the best way to solve the, “Alt Accounts,” problem, and then a second one which goes a bit deeper and asks if the, “Alt Accounts,” problem is really fundamentally a problem at all.

I go into more detail below in the section, “Alt Accounts Problem Further Discussion.”

General Assessment of the Polling Process

Users want flexibility in market timeframes, the ability to freely create markets, fun gaming experiences, but also some safeguards against harassment. Balancing these desires will be an ongoing challenge as one who designs or operates a prediction market platform, but keeping the needs of diverse users in mind will help us build the best platform possible. I look forward to continuing to learn from users as more results of these polls come in before they close at the end of November. These polls are still open and all listed under this group on Manifold.Markets.

Do Long-Term Markets Matter? Further Discussion

One of the comments from a user highlights an important distinction in the discussion around long-term markets. While hypothetical decade-plus markets can serve a purpose for things like guiding long-term policy decisions, actual implemented markets on a platform like Manifold face greater challenges. The inherent unpredictability and changing variables over such long time horizons reduce the viability of these markets for most practical purposes.

The poll results also provide insight. Those who view decade-plus markets as unimportant cite reasons like unpredictability, lack of immediate utility, and opportunity cost. Shorter-term markets may offer clearer actionable insights. As Jeremiah's comment notes, longer-term market positions are easier to liquidate in favor of short-term opportunities.

At the same time, a segment of users see value in promoting long-term thinking, gauging future trends, and encouraging research. Thinking completely hypothetically, perhaps there is some middle ground that may be better served through shorter time horizon markets that connect to long-term issues - for example, a 5 year market on adoption rates of an emerging technology instead of a 20 year broad market.

In summary, while there are theoretical arguments for decade-plus prediction markets, the practical realities likely limit their utility for most users compared to shorter-term markets. Putting my product manager hat on and balancing user needs and platform viability, I would focus development efforts on markets with shorter 1 month to 5 year time horizons and well-defined metrics for evaluation. Exceptions could be made for select long-term markets where clear interest exists. But the majority of resources are better allocated to markets that can provide users with timely, actionable insights.

Motivation of Usage of Prediction Markets Further Discussion

There is a gaming motivation poll that I’m running which reveals an even split between those interested in strategic gameplay/market diversity and those motivated by educational gaming. This highlights twin appeals - intellectual challenge and gaining new knowledge.

Simultaneously, there is a much less successful poll with only three respondents regarding, “aspects of truth-seeking,” in terms of users’ usage of the platform which shows users value acquiring new information and growth through exposure to different viewpoints.

Keeping in mind that gaming seems to be the primary motivation for most users, this suggests many users have a learning mindset and appreciation for diverse opinions, even if participating partly for entertainment. Prediction markets offer a chance to be challenged through game mechanics.

However, the platform itself likely shapes participation incentives. A site focused on scoreboards and competition may attract different users vs. one emphasizing collective wisdom.

Putting one’s product manager hat on for a moment, we might see supporting both motivations is ideal. Game techniques can engage users, while introducing diverse perspectives.

But tensions exist - unbridled competition risks unproductive arguments rather than truth-seeking. Excessive control over topics limits free expression.

One crazy solution may be collaborative gameplay and knowledge sharing, e.g. users collaboratively building markets and marking milestones.

Designing the right incentives and user experience to attract constituencies motivated by growth and understanding in a particular topic could maximize prediction market potential.

Ongoing user research is critical to see if the platform enables enjoyment and education simultaneously. Hybrid models serving different motivations may also emerge.

Balancing Free Thinking While Preventing Harassment

The poll on free speech reveals a preference for maintaining open discourse, though with some guardrails against misinformation and hate speech. Comments from a couple different users highlight the difficulties of defining "hate speech" and how appeals to victimhood can sometimes be exploited.

At the same time, completely unchecked speech risks alienating marginalized groups and devolving into unproductive arguments instead of meaningful debate. A more detailed poll on preventing harassment shows a slight preference for uniform protections over special consideration for vulnerable groups.

Further, even among those who chose that they value freedom of thought in a first-round poll, in a second round poll when pressed fore more detail, currently about 40% of the total responses out of 5 options said that guidelines should be in place to ensure that predictions are based on factual information and not misinformation, hate speech or deliberate manipulation.

Once again putting our product manager’s hat on, we might take a nuanced approach:

• Allow open creation of markets, barring only the most egregious examples (e.g. predictions on private individuals). But provide opt-out tools for those who wish to avoid certain topics.

• Have clear content guidelines prohibiting personal attacks, doxxing, threats etc. But allow controversial opinions if substantiated and debated civilly.

• Empower users to report issues privately to mods instead of public call-outs. Rapid resolution of valid complaints.

• Anonymity options to shield vulnerable populations; though this might be problematic to enact.

• Proactive investment in features that bring users with opposing views together constructively.

• Ongoing training of team members to grasp nuances around speech issues and unconscious bias.

• Seek input from a diverse range of users to improve policies and ensure they do not marginalize.

The goal is cultivating free but responsible speech via guidelines, tools and education rather than top-down censorship. Success will require equitable treatment of users, openness to criticism, and constantly evolving best practices.

Side Note on AI Safety

There are some interesting parallels between navigating free speech and harassment issues on a social platform, and developing safe AI systems:

Just as unchecked user speech risks harming vulnerable groups on a platform, unfettered AI systems could cause harm if not thoughtfully constrained. Yet excessive constraints on AI to prevent hypothetical dangers may also limit beneficial capabilities.

The principles for speech moderation could inform AI safety approaches:

• Allow open "speech" by AIs to maximize capabilities, but with sharp restrictions on clearly dangerous behaviors.

• Set clear "content" guidelines for AI alignment, but don't limit novel or controversial strategies if responsibly executed.

• Empower stakeholders to report issues in private to developers instead of public pressure campaigns.

• Anonymize data where possible to protect privacy; require justification for using sensitive datasets.

• Proactively invest in features that ensure human values are fulfilled by AIs.

• Maintain diverse perspectives in training data and development teams to reduce bias.

• Continuously re-evaluate policies as capabilities evolve to prevent marginalizing stakeholders.

Just as with speech, nuanced guidelines, empowering users, encouraging responsibility, and upholding equitable treatment may be more effective than heavy-handed restrictions in ensuring safety. But this requires ongoing vigilance and engagement with impacted communities. A commitment to safety should be baked into the AI development process from the start.

Could AI Models Be Used to Assist With Content Moderation?

I didn’t really want to get into features as much as philosophy and research in this article, but very briefly - yes, they could be used to flag prohibited content, harmful language, suggest edits, link to relevant platform policies, summarize and distill heated arguments and so on. Whether users would find any of these features useful or annoying, or whether the users would in fact exploit these features in some way remains to be seen.

Alt Accounts Problem Further Discussion

Here are some key points I would highlight on the, “Alt Accounts,” problem related to managing nefarious alternative accounts on a prediction markets platform:

• The polling currently seems to reveal that most users don't see alts as a fundamental issue, but some note potential for market manipulation, trust erosion, and bonus abuse.

• Comments point out platforms that provide free currency are more susceptible - alts allow multiplying finite bonuses. But free currency also helps growth.

• Possible mitigations suggested include limits on new accounts, incentives tied to quality participation vs bonuses, and identity verification for high-value markets.

• However, heavy restrictions could discourage usage - a light touch may be preferable. The culture and market design shape incentives.

• Alt account prevention shouldn't make overly broad assumptions - not all alternate accounts are malicious. Allow room for pseudonymous participation.

• No solution will be perfect - a nuanced, iterative approach responding to actual issues as they emerge is prudent. Extensive upfront restrictions risk unintended consequences.

• The core goal should be aligning incentives. If potential for abuse emerges, target specific actions vs. identities. Promote constructive behaviors.

• Ongoing community input is invaluable. Policies should evolve based on users' needs rather than top-down rules. Trust emerges from transparency.

Overall Conclusion…Thus Far, At Least

In summary, prediction markets face inherent tensions between open participation, integrity, and growth. But focusing on creating the right incentives and targeting harms precisely can help achieve an optimal balance. Moderation should be blade rather than hammer. Perfection is impossible, but responsible stewardship that serves users can earn legitimacy.

Please check out all polls used to create this article listed under this group on Manifold.Markets.