(incomplete) Organizing AI Engineer World's Fair 2024

swyx 2024-06-29

We have just come off a very intense production period of the first AI Engineer World’s Fair, the large, multi-track format of the AI Engineer series of conferences that my biz partner Ben and I are building. I am historically bad at writing down lessons and thoughts, so I am sitting down at my laptop on a friday evening at a McDonald’s with my m2 macbook air on 16% battery and writing and writing until my battery runs out of juice (or I do).

venue

AI Eng Summit was 7-8 October 2023, and went fairly well. I was pretty set on us doing at least 2 conferences a year due to the pace of AI. As a frequent speaker and attendee of conferences I am inspired by large industry conferences like KubeCon (~10k attendees, cloud-native industry oriented) and NeurIPS (~15k attendees, research oriented), by big expos like Big Data London (~4-5k attendees? primarily expo hall with only small talks), and as a business by Infobip Shift (~4-5k attendees, bootstrapped and sold to InfoBip). All of these pale in comparison to -truly- big B2C conferences and trade shows, but they are generally landmark events for anyone in their respective developer sub-industries and it seemed obvious that we should do something like it.

The main problem was that Ben had never organized a conference larger than ~700 people before, and now we were about to go much larger. AIES 2023 was ~550 people; I just named a ballpark number of ~2k people as a goal.

I had always been inspired by the World’s Fairs of a century ago, in my mind the canonical ideal of an expo of the best that the world has to offer, and in San Francisco if you have seen the Palace of the Fine Arts, or the Space Needle in Seattle, or the Eiffel Tower in Paris, or the China Pavilion in Shanghai, you’ve seen artefacts of former World Fairs permanently leaving their mark for modern cultures in a pale imitation of how ancient civilizations built megaliths and ziggurats for us to marvel at. So I also named the Palace of the Fine Arts as our ideal venue, and we launched the World’s Fair -at- Summit itself, with “blind bird” tickets - fully refundable, no dates set yet beyond vaguely “Spring 2024”.

The venue immediately turned out to be the sticking point. We did site tours of PoFA immediately after Summit. It had a beautiful theater that could fit ~1000 people, and a hall that could comfortably fit the expo and some stalls for food booths and food trucks. I thought PoFA could work because HuggingFace had hosted a ~2,000 person “Woodstock of AI” there a year ago and people seemed to love it. The Rotunda of the PoFA was a nice outdoorsey venue where we could hold a picturesque reception and perhaps open air concert (I was friends with a violinist who would love the opportunity to bring a string quartet). For the lake and park around the PoFA I even dreamed of having outdoor stands and booths and people flying drones around.

The reality was that PoFA is heavily regulated by the City of San Francisco. Outdoor stands would require extraordinary approvals we’d probably not get as an unknown conference. The Rotunda had historical significance and a very heavy extra cost to rent out. PoFA itself wasnt cheap but the real concern was noise leaking for a multitrack conference - we were looking to have 4-5 tracks, and our choices were smaller rooms of between 100-300 in capacity, that 1k person theater, and perhaps 1-2 areas where we could fit 300-500 people with lots of background noise. Because it was out in the Presidio we would also have trouble finding a hotel for guests and speakers to comfortably get to/from, and might have to charter buses. I was fine with it but we ultimately decided against it in the end.

By end January we had ended up going for our backup option, Fort Mason, which was set on the piers and had great views, and had the outdoorsey element I loved. We announced a Save the Date in end Jan-early Feb with a few visuals, and that helped kick off our early sales.

However, Fort Mason also didn’t have enough stages for us to do multiple tracks, and we fretted about the cost going into March, because if we hit the low end of ticket sales projections it looked like we would be on the hook for very, very high fixed costs to the tune of 300-500k in losses, effectively bankrupting our business.

Venue needs to be played off against scheduling - I had my heart set on a Spring conference, and a 6 month pace would put us in April, and the concern was both cold weather and availability of the venue in short notice. I dont remember all the constant back and forth but we pushed and pushed until we ended up at a June date.

We sold “Blind Bird” tickets with no fixed venue the entire time - we had soft holds on both venues but could back out based on ticket sales and our other options. very late in the game (March?) Ben ended up finding the Marriot Marquis through an old contact of his, and eventually fell in love with their flexibility, ideal AV situation, site in the City, and the View Lounge that overlooked the whole city. I hated the idea of running the conference in essentially the air conditioned basement of a hotel, just like a billion other conferences before us, but Ben was enthusiastic about it and I gave in (it turned out not really to be an issue as far as I can tell, but I still maintain that outdoors-ness is more “fair” like and would have made for better social media).

With great venue requires great sponsor. I dont know if we can talk about the sponsor search and discussion in great detail, but our dream presenting sponsor was -always- Microsoft - they had supported Summit, were happy with it, and now we were about to ask them to trust us a lot more. They eventually agreed to do it, thanks to great champions from Britton on down.

launching

We launched the site with Msft Deputy CTO Sam Schillace on March 29. We had soft commitments to speak by then from all the major AI labs we wanted to feature (see below for more) so we were reasonably confident this would be a draw. but apart from venue and presenting sponsor (and a few others) we had basically nothing else locked in with 3 months to go - i did of course extend verbal offers to speakers i was courting for most of the prior 3 months.

By then I had also decided on the 9 tracks we wanted to feature, which led right into…

speakers

“I’m incredibly curious how you got the buy in of nearly every influential person/company in this segment of the industry (in the US at least)”

It seems a lot of people are interested in how we got speakers so I will recount as best I can… but the answer is going to be a bit boring. Just grind, make LatentSpacePod, have track record with AIE Summit, and ask people to come that you think will be interesting to your audience.

longer form:

We put out a Call For Proposals together with the launch, a traditional application process.

Amazingly, we got some incredible speakers INBOUND simply via this process, including 4 of our keynote speakers (Stephen Hood of Mozilla and Chris Lattner of Modular, Ishan Anand of Spreadsheets are all you need, and Danielle Perszyk of Adept - who eventually dropped out because of Adept’s acquihire around the date of the conference), and many high profile/worthy speakers that we accepted straight into the conference:

João Moura - crewAI, Founder & CEO
Eno Reyes - Factory, CTO
tevye Krynski - Perpetual, AI (still in private beta) CEO
Patrick Debois - Jedi, VP Engineering
Two speakers -: Hamel, Husain & Emil Sedgh Parlance Labs (Hamel), Rechat (Emil) AI Engineer (Hamel), CTO (Emil) (invited by swyx)
Nikhil Thota - Perplexity, AI “Member of Technical Staff”
Ahmed Menshawy - Mastercard, Vice President of AI Engineering
Arjun Bansal -, Alexander, Kvamme (ak@echoai.com) Log10, Echo AI (Joint presentation) Log10 CEO & Co-founder, Echo AI CEO
Philip Kiely & Pankaj Gupta - Baseten, Head of Developer Relations, Co-Founder
Jeronim Morina - AXA, Senior MLOps Engineer
Ian Webster - Discord, Senior Staff Software Engineer
Shawn Jansepar - Khan, Academy Director of Engineering, Learning & AI Platform
Dominik Kundel - Twilio, Head of Product & Design, Emerging Tech & Innovation
Daniel Whitenack - Prediction, Guard Founder and CEO
Atita Arora - Qdrant, Solution Architect
Mark Moyou - NVIDIA, Sr. Data Scientist
Vibhor Kumar - Tinder, AI Software Engineer - Trust & Safety
Sumit Agarwal - Gather, CEO and Co-founder
Remi Louf - .txt, CEO
Phlo Young - Young, Marketer
Morgante Pell - Grit, CEO
Shelby Heinecke - Salesforce, Senior Research Manager
Emil Eifrem - Neo4j, Co-founder & CEO
Arvid Lunnemark - Anysphere, Co-founder & CTO
Jasmine Wang - LanceDB, Head of Ecosystem Enagement
Mark Kim-Huang, Gradient, Cofounder, Chief Architect
Sheila Gulati - Tola, Capital Managing Director
Vaclav Mlejnsky - E2B, CEO
Gunjan Patel - Palo Alto Networks, Director of Engineering
Phoebe Klett - Normal Computing, Machine Learning Engineer
Stefania Druga - Google Gemini, Research Scientist
Hubert Misztela - Novartis, AI Researcher/ Director Data Science
Manuel Odendahl - The Tree Center, Principal Engineer
Tanmai Gopal - Hasura, Cofounder & CEO
Banjo Obayomi - AWS, Senior Developer Advocate
Aaron Dignan - Plumb, Founder & CEO

These ~30-something folks were just ~6% of the ~500 that applied that passed our review/selections.

That meant at least about 30-50 more invited speakers that was entirely my job, apart from speakers that would be supplied from our sponsors.

I established a target list of dream speakers per track that I laid out, then set about inviting those people. I probably cant show you the list but basically imagine 8 candidate speakers x 9 tracks and me just spamming out every credible/worthy name I could think of. The biggest problem with doing 8 candidates was I didn’t actually know what my “inventory” was - how many slots i had to offer - until very very late. something I probably should have agreed earlier with Ben, and caused a STUPID amount of undue stress and sorting trying to curate the right schedule while also not sending out invites that I would then have to retract if I overcommitted in any one particular track.

The most challenging were keynotes - we were variously trying to book Jeff Dean or Aidan Gomez or high up Microsoft, Anthropic, HuggingFace, and OpenAI person but their travel schedules were already fixed for those times. Some folks I reached out to initially accepted, and then became unresponsive, which caused extra chaos.

AI conferences need AI labs to bring the spice. We had OpenAI and Anthropic participation last year, so this year I needed the rest - Cohere, Mistral, etc. We got those thanks to their devrel and VC contacts. I made glaring omissions with Meta and Huggingface - which I basically have no excuse for but for just being stretched too thin.

AI conferences also need hot AI startups that poeple want to see. As you saw above we got decently lucky with our keynote CFPs already. Belle of the ball was Scott Wu/Cognition. Fortunately he had heard of me and we had chatted once - I just offered it and he accepted. not much to say there. I did similar opportunistic offers around Cursor, Cartesia, Dawn Analytics, Midjourney, Zeta Labs, Groq, and the 6 authors of the Year in LLMs post. Having an eye for what is “hot” and simply asking I think did most of the work here.

One thing I really tried to do was make the World’s Fair representative of the world - I got contacts for African, Middle East, Indian and Chinese AI companies, and also invited contacts from Singapore and Japan. Basically, none of them bit, except for the Chinese groups like Qwen, who were interested but needed a 6 month lead time for visas. So basically we just got Europe and Latam in the end.

The other more challenging thing to handle was the AI Leadership track - something I had never organized before, a persona I wasn’t particularly close to (and to a lesser extent the AI in F500 track as well). It turns out that I need not have worried - I attended an AI event by SourceGraph who gave me a lot of inspiration as to what appeals to AI Leaders and they helped me book a few people, then my natural network and the CFP (above) brought in the rest very organically. My proudest moment was inviting OpenAI, Cohere, LlamaIndex and Hex to do special sessions focused on gaps that I felt were missing, and they all accepted. Based on reviews, the AI Leadership track went particularly well.

One feature I created very late in the game was the concept of Track Hosts. I saw that Ben and I would be so stretched that we would never be able to ensure a good experience of the 5 simultaneous tracks, and having tracks without emcees felt bad. I again just rustled up my network to get the hosts together and most were happy to contribute, I think every single person I asked accepted (some actually preemptively volunteered, like Demetrios and Peter Cooper)

A final word on diversity. Last year we had 10% female speakers. This year I believe we had 50% female keynote speakers and track hosts, before the last minute changes we’ll talk about next. Part of that was me putting my thumb on the scale slightly, but the inbound pool was genuinely more diverse and that of course organically helped the most. We topped it off by having my friend Sasha Sheng run our first ever Diversity committee, which yielded ~35 diverse attendees of the conference where we handed out scholarships, partially graciously supported by Writer who were the only diversity sponsors we managed to get.

the week prior

this is not precisely the 7 days leading in, maybe more like the 1-2 weeks leading in but i’m just recollecting the very very high stress period ramping up into the conference

Marketing:

May 17 - announced first wave of speakers
Jun 3 - second wave of speakers
Jun 17 - fleshed out VP AI sessions and side events
we also advertised on major newsletters from cerebralvalley.ai to latent.space and worked out media partner deals with some newsletter authors. not sure the ROI on these.
We started feeling the sales pressure flip from undersold to oversold around Jun 20, and I put a successful call out for volunteers that then led to many folks incl Andrej Karpathy signal boosting 2-3 days before the actual event.

We aimed for ~2k attendees; on May 1st we had sold 25%, Jun 1st we had sold 41%, and only sold out on Jun 24th after Andrej’s tweet. Here are some intentionally disaggregated line charts of the aggressive last-minute-fomo sales curve that make events like these extraordinarily hard to plan in advance:

For comparison observe how many years it took ICLR for their sales curve to reach an ideal 50% of tickets sold 60 days before the conference.

Content:

Many, many things are decided last minute. Every speaker is an external dependency, and every speaker themselves has dependencies we don’t know about that could sway their commitment at any time, so its better for them the later we lock things down. A lot of ideal people - including OpenAI, Anthropic, Cohere, LlamaIndex, HuggingFace, Supabase, Exa, and others - only came inbound to speak last minute and we variously were either already full out of space or had to use them to backfill dropouts. And of course for our attendees and venue/printer partners they had the opposite need - everything should have been confirmed weeks earlier to be better.

This is how rough the schedule looked on Jun 17.

non-exhaustive list of things we did leading in…

finalized and sent instructions to track hosts
applying for visa waiver, getting rejected, and having to back out of multiple in person commitments I had made going into the conference, as well as my in-conference commitments esp with GitHub CEO
booked socials interviewers (for showfloor walkthroughs and video marketing - we had 3 groups doing interviews)
booked external startup partners (Bee and Compass wearables for speakers/track hosts, Videotap, Thoth for infographics)
speakers pulled out last minute for workshop - had to reschedule
fielded requests from sponsors on sending list of attendees - we do not sell attendee lists
confirmed livestream tracks
confirmed new speakers and their timings
dealt with very public criticism of AI Engineering
prep World’s Fair 2025 material
constant bickering over our internal tech and processes and onsite presence balance of sponsors and speakers lol
finalize 100+ person speaker dinner and swag logistics
ramp up with ~50 volunteer team

what we did well

lessons learned

internal organization
- people don’t show up to things - even when they have RSVP’ed yes, even when they have paid for it. we could comfortably book 130% of our capacity and still be fine; this time around we erred too much on putting hard caps at 100% and stressed ourselves out for no reason
- must have single source of truth - we had up to 5 spreadsheets tracking speakers and room and time assignments and everything was manually being written up. AND those all were different than our CMS.
- must have all speaker and track host phone numbers - was very stressful when we needed them and could not find them (led to one keynote speaker missing their talk completely)
- must offer preregistrations for sessions/workshops?
tech (mobile app and website)
- load test database/CMS - likely to go down most when we need it. in our case we had 4-8 hours of outage because we didnt realized we hadn’t switched to connection pooling with supabase
- mobile app was slow to load
- webapp slow to load - until we switched to pregeneration halfway through
- we added mobile app features for build your own schedule, matching, and lead scanning - TBD on what numbers we got actually using the app
- offer API to our schedule so as to enable others to build on top, because thats exactly what they did in frustration with our website/app
content curation
- no prerecorded demos from bigco please
- allow more room for social events - Weaviate wanted to do morning yoga, other guys wanted to do run sessions
- accept much more online talk submissions - we had a LOT of dead air on the livestream

length and precision

Tiago Freitas — Slots should be longer, they were too short for the speakers to do anything serious and everyone I talked to complained of the same

Eugene — Lots of folks missed the more hackerish nature of the previous year, the wilder the weirder ideas the better? - so maybe have a random track? - people like seeing more random folks like them trying stuff (i wasnt here last year, but heard this repeated)

Slightly more free space to hang out, SaaStr is an example of optimizing for hallway communications by providing more table and chairs for folks to hang at (im bias, cause thats all im here for)

Some multitrack conference, usually have a big board in the center with all the tracks schedule. All in one spot - It can get confusing otherwise