Facebook M is part digital assistant, part experiment in artificial intelligence. The recently revealed service runs atop Messenger, Facebook’s existing instant messaging app, and it’s billed as a tool that will do things even the best-known assistants, Apple Siri and Google Now, can’t yet do, like make an appointment at the DMV or buy a gift for a friend. The trick is that M is driven in part by human operators, but the hope is that in tracking the behavior of these humans, Facebook can build an artificially intelligent system that behaves like people would.

The question now becomes: How well does the service work today? Late last month, Facebook released the service’s initial incarnation to a few hundred people in the San Francisco Bay Area, but the company hasn’t provided a public demonstration of the service or shown it in action to reporters.

This past week, however, a Mountain View, California-based company called UserTesting poinpointed an M user among the network of consumers it utilizes to test all sorts of tech products. The company invited the user—whom it declined to name—to run the service through a fairly broad range of tasks, and it ended up recording these tasks and tracking the user’s general opinions on how things played out (see video below).

According to the company, the user described the experience as “surprising, useful, slightly creepy.” M could perform multi-step tasks—making restaurant reservations, looking for new deals on local internet services, and providing recommendations for weekend getaways, among others. This indeed put it a step ahead of purely virtual assistants such as Siri, the user said. But he complained that the service was too slow to respond to his messages and said M couldn’t complete some tasks in a reasonable amount of time.

“Our test participant was impressed with how much M could do, but was sometimes disappointed at how long it took,” UserTesting’s report reads. “He concluded that it would be very useful if he could set it to perform a non-­urgent tasks for him while he worked on other things.”

This is just one set of tests with just one user. But it shines a light on how difficult Facebook’s task will be. (Facebook declined to comment for this story). In order to collect data for a more advanced digital assistant, the company must ensure that the existing service works well enough—and that people use it. “This is a real intriguing idea,” says UserTesting vice president Michael Mace. “The trick is that with a mobile device, people are conditioned to expect immediate responses. It’s the era of instant gratification and the smartphone is the device of instant gratification. One of the interesting challenges for Facebook is: how do you manage those expectations with services that are inherently going to take some time.”

Timing Is Everything

Before the launch of M, Facebook’s Alex Lebrun, an AI guru who works on the project, said that M could handle everything from telling jokes to planning weekend getaways. The tester tracked by UserTesting asked M for a joke and wasn’t all that impressed. He gave a it 3 on a scale of 1 to 5, and the service needs five minutes to tell it. He was more impressed with its ability to recommend a restaurant—and properly understand what he was asking—though he felt the task took far too long (11 minutes).

“The user appreciated that M understands the context of the conversation, so he could simply ask it to ‘make a reservation there’ and it would understand where ‘there’ was, unlike Siri,” the report says.

The weekend getaway? This was a bit more complicated. M took so long to complete the task that the user eventually gave up. After 15 minute wait, M provided an article listing 22 of the top getaway spots in California—something the user could have found with a Google search. “Though the results took a long time and were disappointing,” the report continues, “the user acknowledged that M could be helpful for this purpose if he could just let it run in the background.”