E203 - Challenges of Flying a Helicopter on Mars

| Disclaimer: Transcript provided by OtterAI and YouTube automatic Closed Caption. Any inaccuracies or errors are not attributed to the Hosts or contributors to Human Factors Cast. |

hey everybody what's going on it is April 22nd 2021 uh my name is Nick Roome and this is human factors cast i'm joined today by mr Blake Arnsdorff over there hey Nick how's it going man it's another wonderful thursday hey blake it is a wonderful thursday you know what we got going on for uh programming notes here if you haven't already um there is a uh there's a healthcare symposium coverage that we've been doing over the last week man it feels like we got a lot of stuff coming out so elise and i have been at the conference all week um we're we brought you a recap uh it's in your podcast feeds already we did that last night um there's a couple more interviews to come so stay tuned for those there's one already out with rebecca butler um who is who works at medstar health definitely worth checking out some of the interviews we're bringing you a lot of different coverage um so like i said more interviews to come it's a great time go listen to them if you haven't already uh but we know why you're here we're here for the human factors news let's get into it

that's right this is the part of the show all about human factors news this is where we talk about everything related to the field of human factors we could be talking about supervisory control we could be talking about simulation we could talk about the design process a little bit of everything as long as it relates to the field of human factors that is fair game for us to sit here and talk about blake what do we got up this week so this week nick there's helicopters now on mars so recently the mars helicopter ingenuity took the skies of mars for the first time and it did so fully autonomously so this ingenuity helicopter was designed for autonomous flight out of obvious necessities since the time delay between ingenuity's pilots at the jet propulsion lab or jpl and the jazzero crater on mars makes manual or even supervisory control absolutely impossible so the best folks at jpl practiced and did this all through simulation and then they were the hope was they would be able to handle the helicopter and everything from testing and simulation here on earth and here on earth simulations are very critical for things like this so it's a critical tool for many robotics applications because it helps you not have to rely on using expensive hardware potentially breaking it and you can just do stuff in the in the moment and test and solve problems and once you think you get everything right in a simulation you can then go and actually test it out in the real world but in this case you're testing a helicopter on mars so it's a little bit higher stakes testing the mars helicopter under conditions that match it and you wolf they found were just not even possibly replicable on earth and jpl has flown a bunch of different engineering models in martian asmr atmospheric conditions and they actually ended up using ac actuated tethers here to try and mimic some of margin's gravity but there was really no way to know how the helicopter would perform outside of simulations until they actually float on mars so with that in mind the engineering team has been heavily relying on simulations since it's the only tool they can get to actually give them a perspective of what it's going to be like trying to control this thing or autonomously fly it so nick how in the world has this happened this is just nuts to me that we've put a helicopter on mars that could be autonomously flown and provide us footage back here on earth it's nuts right like you and i were talking about this before in the pre-show that it's kind of nuts that yes we're on mars again we've been on mars we're on mars again and it's so weird to see stuff going on on mars and now there's a helicopter involved um so so yes uh i kind of um buried the lead there at the top when i said supervisory control simulation and design process we're going to be talking about all those three in relation to the um helicopter that is now on mars uh but if you're just asking for my general reaction yeah it's freaking nuts man this is cool uh what do you think about your mug helicopter on mars so the thing that really blew my mind is like thinking about the fact that we've got a rover up there that's able to take a lot of high quality video of this thing actually functioning on mars and when i thought about it and looked at the design it's such a smart and sleek design and of itself that it just kind of blew my mind that this thing was even able to fly in such a tough atmosphere and that the doubly so and i think we'll talk a little bit about this as we go through kind of the deeper human factors related points but the fact that you would have to rely on testing something basically only in simulation and you're gonna fly it autonomously that just has such far-reaching implications like as a as a human factors person as somebody that worked on it that would be working on one of these teams i can only imagine the amount of kind of like well we've done as much as we can on simulations on earth but really we just have to put it out into the wild on mars and see how it performs that's a giant leap of faith in some ways yeah and i mean they did do things here on earth to prepare for that um you know we can talk about those uh with with simulation and the design process they made sure that they got it right here before they moved it over there and there's still some guesswork involved um let's start with the supervisory control aspect of this because this is a unique situation where you can't really have supervisory control of this helicopter right i mean that's how you would do it normally with a drone um here on earth what you would do is you would have a drone you'd be controlling it and you'd be able to react in real time to what's going on um and so you know you have sort of um this reactionary piece that you can't necessarily do from afar uh it's a totally different game because and and the reason this is is obviously the the time difference that it would take to receive that feedback the drone would already be on the floor um and so like i don't know i thought it'd be kind of cool to go through some of these supervisory control um tenants of human factors if you will and explain why they are not you know able to be applicable here in this situation um blake what's what what you got so at so the way to kind of think about this like sheridan laid out a long time ago i think in the late 90s this is where this paper is from but anyhow there was like three main problems that exist just with super like the implementation of supervisory control on top of adding automation to it that we're gonna really affect how people are able to maintain any kind of supervisory control but before we dive too deep let's do the best scientific job we can to kind of operationally define what we're talking about when we say supervisor and control so that's just a general term for control of many controllers and control loops so imagine a uav that needs to be able to be past control back and forth between multiple different entities or multiple different controllers so somebody's flying it at one point they need to hand it off to another controller or somebody can request control of it that's the kind of like simplest definition of it and you can imagine that adds in a lot of different potential issues you know you know losing losing control of something or line of sight problems or even mitch matches in an understanding of you know did i hand this thing off to another controller there's a lot of things that can go wrong now let's kind of like add automation on top of that can you imagine just issues that would be running rampant just from multiple people trying to control one mechanical object and then if you put automation in the loop it adds kind of another layer of complexity so when we think about this in terms of putting something now into space where the latency is going to be so bad or so impossible to really understand you could see how supervisory control just from a basic definition is going to be nearly impossible if just not even a good thing to even try yeah i mean you talked a lot about the the different controllers that go into this the different systems that are controlling something automation different humans uh different operators there's also the control loop right which is the the process by which i was describing you you see the feedback or you get feedback in some way from sensors on board um in the uav example you would you know see an altimeter um you would have a rough approximation of what your you know where in space you are right there might be a camera on the drone where you're able to take in information as the pilot and you're able to react to that information right and and the biggest thing with this um helicopter on mars is you don't have a control loop if you i mean you do but it's the time delay between when an action is taken and your reaction to that is so great that you would not be able to correct anything that needed to be corrected um and i mean they do have control loops when it comes to like the rover right they can move it um you know inches at a time if they need to get out of a precarious situation but in a situation where you have the helicopter you can't do that because it's flying at such a velocity or it's it any kind of critical um mistake could uh cost you know your it could cost nasa a helicopter let's put it that way right because if you think about it you know if let's say there is a rock that the rover can't get around well that's not that big of a deal right it's stuck on a the wheel got stuck on a rock well they have to back it up and move it and go forward and that can be done in a loop where they can understand exactly where the rock is and there's no consequences because there's no immediate threat right where a helicopter you have um the threat of potentially you know if there's a uh there's no gusts of wind on mars right like that was one of the biggest yeah that's true so so um the the big thing with that is that you know it could just fall flat and and land incorrectly and you that's it you know i mean does the ingenuity have an arm that they can go over and like try to prop it back up i don't know i'd imagine but um so so that control loop just in general is completely gone you don't have that with that's something that you have in typical supervisory control um you want to talk about these three uh aspects here of supervisory control as well absolutely yeah so these are like kind of this is one of the areas that i think is really interesting from past work on supervisory control so the big thing that you've kind of talked about if we don't have the control loop and even if we did we're going to just experience this massive loss of under of the a person's mental model of making an interaction through a piece of software trying to control something that's you know millions of miles away um and the fact that that one for one interaction is not there so you're just gonna lose somebody's ability to understand or be able to kind of even do mental calculations on behalf of the machine of like what they could do um in terms of interaction so you lose out on somebody's mental model and kind of being able to align with it but i think the biggest kind of thing to talk about here is this is an instance where you have to like it as a human factors professional you and definitely in grad school i heard a lot more about kind of levels of automation and figuring out as as you scale automation what that means for an operator in the loop and something in this case is that this is that top tier of fully automated systems because there's just no way that a human operator because of the parameters of what we're doing the distance the the lack of communication and all that kind of stuff without a without like full autonomous control and simulations of that autonomous control you would not be able to kind of accomplish the things you need to or that you're hoping you can do with this helicopter or putting anything else on mars for that matter so this is one of those instances where just like the cost benefit analysis in this case like from the human factors perspective leans much more heavily in terms of automation leading you to need to do these kind of simulated tests yeah those are great points um you know i think i think that's right there's the automation that's going to have to do the heavy lifting here humans don't really have that much control in this situation um and you know i i think i don't know it's it's hard to for me at least to apply these principles to this situation because they're just absent right i mean you talk about a traditional uh control loop where you get that feedback back and you do get all these um principles right of of uh supervisory control you get all of them and so uh but with this rover or with the helicopter at least you don't um you know anything else to add for supervisory control because i do want to get into the simulation bit because this is where a lot of the magic kind of happened with this i think let's move to the simulation because i mean this is how this whole endeavor was even possible was being able to both simulate things in real life here through the design through different models that jpl had put together so where would you like to kick off with simulation for this particular yeah well well let's just talk about simulation in general um simulation in general is a way for you to um estimate something without serious cost uh to the actual thing right so think about like flight simulators this is where pilots fly panes in a simulated environment so that way if there's any critical errors they're done in a simulated environment they get that practice for the real thing where hopefully they won't commit those errors or if anything goes wrong they'll be able to react appropriately so the advantage of being in a simulator is something where you know you can um you can certainly uh react to things that are uncommon and that way you have that built up muscle memory for those things that are uncommon and you can react appropriately when it actually does happen in the real world um there are uh just a side note really quick there's a whole simulation track um that they came out with in the healthcare symposium this year uh i attended some of those talks it was very interesting so if anyone uh went to that conference go check out some of the simulation chats they were very cool um breaking down why it's so important to have at least that simulation aspect in a hospital you can imagine but it's there um so let's talk about the simulation environments for this helicopter and what they did um so they actually developed this over at jpl um they looked uh at a system basically um that they've been using for a number of missions i think they said uh 30 years so it's been around for a while but they took that framework and kind of built their own helicopter simulation based on that framework um and they used their own rotor models so this is kind of getting into the technical bits of it so this is more or less how the helicopter will react in that environment based on this model right so there's uh they have the aerodynamic models that feed into this simulation and a lot of this at this point sounds like technical babble right this is all different models that fit into the simulation if you think about it like a if you think about a flight simulator there's these um air simulation models that uh will make the plane react in different ways right so the more accurate you can get to how air actually impacts the vehicle the plane at that in that instance uh the closer you can get to what it's going to be like in the real environment so they try to model that with these frameworks um and then so you know the the whole question of whether this is valid and the validation bit of this simulation what you know what does that look like um any any thoughts so far blake before we get into the validation i mean although this is like this is a lot more of technical sides of things it it really is interesting because it informs the entire design i mean without this kind of for this formal testing and giving you and it also kind of pulls on jpl's past knowledge of already having stuff that's deployed on mars gathering information in terms of weather and how the atmosphere interacts and so being able to actually simulate it before you start building or even start trying to build you know simulated models that act like mars's um atmosphere and what the helicopter might experience i mean it cuts down on cost and from like a just like a human factor's engineering perspective the capability before you even get to building anything have an idea of what design is going to work best based off of just early simulation and modeling that's already existing through jpl's kind of um either models and designs they've done in the past so it's it's kind of just a awesome feat of engineering in general um but what it what really did they do in terms of when was enough in the simulations in terms of validation yeah so that's that's a good question so you were talking about the design process and we'll actually get to that process in a minute um and it's very interesting how this simulation and that design process kind of intermingle and so what they basically i mean to them it was like close enough uh for them it was what did we understand enough about this uh problem space um to where we can actually test this then what they did was once they had enough information for them um they actually went on to test the simulation the like virtual simulation in a physical space um they did it in a vacuum vacuum chamber so they were flying this helicopter in a vacuum chamber and uh they were replicating sort of the mars atmospheric conditions within that chamber um and so you know these tests were you know done before they actually tried to fly the helicopter they were looking at um sort of the system identification which is looking at the properties and dynamics um and how how the thing in the physical environment actually reacted to the uh environment versus how the thing in the virtual environment reacted or with the models reacted to the environment and what was the delta between the two how is the physical environment different from the simulated environment and how you know can we use that information from the physical environment to update our models and produce a better simulation um so they did this in several stages you know they're doing things that they can't necessarily fully replicate until you do this in the real environment and so it was a lot of going back um back and forth and one of the biggest challenges for them was kind of this uh the martian gravity you can't you know we they can simulate a lot of the same stuff here on earth yeah environmental conditions they can you know the everything will work exactly as it does except for gravity gravity is different and so you know the way they try to get away with this is um attaching a little string to the top of the helicopter and and pulling up on it so that way it reacted a little bit you know lighter to uh earth's gravity than you know to kind of simulate the mars gravity so it's not perfect you can't right but yeah it's not perfect but that's that's how they did it right and so that's kind of where they were coming from with this validation process um you know and they were talking about what you can do to compensate for the physical testing that you can't do on earth uh that's kind of the the um the string on top of the helicopter that i was telling you about to pull up to simulate mars gravity which i thought was a kind of a clever um uh old-fashioned solution right absolutely yeah just tie a string to it it'll be fine yeah and then so the the question came up right whether or not this was um are they relying on simulation too much um but it's their only option so it's what else can you really do at that point yeah you have to keep testing um so i don't know do you have anything else to add on simulation there's there's a couple more bits in here about um when did you know how far did you push the simulation and um what they learned from it so we can get into those but did you have any thoughts at this point blake on the simulation aspect of things i guess what what i found interesting that i i wouldn't have thought of because when you say helicopter i really just think about like okay it's just it's got to lift off the ground there's going to be a lot of kind of constraints that we like you said we can't necessarily simulate to a t but we can add crazy strings to it or we can like push it into you know these chambers that should be able to at least give us something close to what we're modeling in our simulation program side but then there's the the other aspects of this thing because it's not just meant to be a helicopter it's got to be a sensor it's got to have sensors on it and have cameras going as well so like trying to test that stuff to a point where you feel like you've tested it and pushed it as far as you can um and then feeling confident that like okay we we've developed sensors and camera equipment that we think will be able to survive you know this kind of harsh environment that we can't do anything else to simulate it in terms of doing it on earth or doing it in models through programming it must have been really interesting to kind of have to figure out like okay sensor wise what do we need and what's the maximum we can really push it to so that it can do something different than what the rover's doing in terms of what it's gathering so the the limitations of simulation here i mean at some point you do just get to this place of it's it's as good as we can get it now we really have to make that jump and actually push it out to test it on mars itself yeah we can just throw another helicopter on mars if it doesn't work and that kind of goes into the design that kind of goes into the design process so why don't we jump into that a little bit because this is very different from like a design process that you might have here uh for designing a product in uh on earth um because you know in that iterative design process you would have a build test update repeat right that you'd have that kind of framework where you would test something you'd go out to the backyard test the helicopter um but they can't do that here and so um basically the engineers were saying they had to get as close to the final product as they could in that first iteration before they built it because um you know obviously the first iteration is not what's sitting out there on mars now but they got close enough to where they could test a prototype and understand the wind dynamics or the aerodynamics i guess of mars um and then they updated their design based on that i'm not sure exactly how many designs they went through before they ultimately sent the uh the final um prototype out or the final thing out right it's not a prototype um but yeah i just found the design process and and the limitations on the design process very interesting for um you know for designing for something on mars right that's kind of crazy yeah because i mean the i'm so used to just thinking about the software side because that's really where i think i've spent my most time i really don't think i've had too much impact on hardware itself but i could imagine it being a little it would feel hamstringing because even on other things you developed maybe through jpl in the past it you could you know go to creating actual physical models or rely on stuff that you've already created before or maybe sent to other planets potentially but even then i wonder if the the iteration was a lot to do with or had a lot of benefits because they were doing so much virtual simulation so if you're able to do things in that regard and kind of like take models that exist of you know other types of aircraft whether it's helicopters or whatever they started with and really testament virtual environments if the kind of like design really fleshed itself out there digitally before they even moved to worrying about the first initial build of this thing but at the end of the day like you're saying you you would have to build it and then test the physical aspects of it because there's only so much at this point that a virtual simulation is even going to tell you until you get that first iteration of materials for that first hardware product yeah so i it would be kind of irresponsible of us to not talk about the control aspect of this thing how does this thing actually get controlled um and the way i understand it or the way that this um this chief pilot of ingenuity is talking about it is that they have to pre pre-program um flight paths into the helicopter so it will kind of do it based on uh the the operator's sort of flight path that they send to the to the helicopter and they're starting small right i mean we've flown a helicopter on mars but what that flight consists of is a three meter rise and a three meter descent um and so you know that's that's really what we've done at this point but to take off and land and test to make sure that the aerodynamics do in fact uh you know check out um where that's not gonna be critical if it falls down because it'll just land and you can try it again uh hopefully you'll have another shot if you just go up and down uh without you know eventually they would like to take this thing out and survey like you said blake with the cameras and sensors and everything um and that's going to be a very different flight than up and down so um you know talking about the controls and how that interacts with the design process you know there's a bunch of different parameters by which you have to control it's not as simple as like just using a joystick no you have to specifically um program in exactly what you hope the helicopter will do right so for example one thing that wasn't included in some of the initial designs was um the cyclic control on the upper rotor uh they didn't have that and they found out that they needed that based on some of the simulation and models and prototyping um they found out that that's a control that they actually needed to have control over because it's pretty important for um you know uh the disturbances that they thought they might see on mars uh from the aerodynamics perspective so they needed to have that extra control authority over the um upper rotor as well so which you know so that was done mostly in simulation but you know they they kind of see um they saw that in the prototype the prototype only had the bottom control and so they realized they needed it and it's kind of cool to see exactly that relationship right they test these things and realize oh we need a control for this thing because it didn't exist and that is something that could really impact the the flight path or the flight dynamics of this um helicopter that yeah that's very true and when they try and even move forward with this thing i feel like because the the vertical liftoff in landing that makes total sense as a starting point but i know that the terrain can be kind of very different right on mars than anything you would be able to even simulate realistically because that was a constraint they mentioned about the simulations in general especially in how it impacts sensors on top of just the rotary components so going from here i mean basically it's it even though they've done a lot of simulation and testing i can imagine that first kind of mapping out a set of waypoints for where they're going to try and fly this thing and do its first survey that's really going to probably yield a lot of different information potentially about how changes the design will be made or to help kind of inform the next kind of iterations of this thing because although ingenuity is definitely the first of its kind like they even do mention throughout this that there's already some in the works for larger helicopters that can do a little bit more carry more sensors or even can break down into various components um outside of just being a singular unit so it's it's just mind-blowing though the stuff they're able to like come up with and the fact that we're able to although we're not you're not directly under any kind of supervisory control through autonomous programming we're able to collect and transmit along with you know fly things on a different planet and be able to relay that kind of stuff back to continue feeding you know different models and stuff like that for the atmosphere for collections whatever it may be yeah you did mention kind of the future here right we could talk about the next step so what they're doing now is these flights on mars will actually send them pilots and engineers data back which they can then feed back into their models with real world real mars world data that will then help improve some of the simulations that they can run here on earth and with that more data they can understand basically how their flight paths might be impacted by real mars world data and so it all feeds back into itself right the more flights this thing gets on mars the more data they have to feed into their models that they can use to fly this thing better on mars it's it's all kind of cool how it all feeds in together and i love it um there's your actual feedback loop right there yeah exactly except where it's very delayed and anyway uh anything else to add to this wonderful news story about flying a helicopter on mars i i'm just blown away by it i've really loved seeing some actual footage of this thing flying on mars because that just was so much it it brought this home so much further because they do if you haven't checked out the article that we posted on both the blog in the description of the podcast everything like that it's worth checking out because they do a breakdown of the design and development of this thing as well as you get to see a few shots of this thing flying on mars which is just so phenomenal yeah i was speechless when i saw it i was just absolutely so cool to see all right well thank you to our patrons this week for selecting this topic and thank you to our friends over at ieee spectrum for our news story this week if you do want to follow along we do post links to the original articles uh in our slack as we find them and we also post them on a a little news roundup every week so we're going to take a quick break and then we'll be back to see what's going on in the human factors community right after this

human factors cast brings you the best in human factors news interviews conference coverage and overall fun conversations into each and every episode we produce but we can't do it without you the human factors cast network is 100 listener supported all the funds that go into running the show come from our listeners our patrons are our priority and we want to ensure we're giving back to you for supporting us pledges start at just one dollar per month and include rewards like access to our weekly q and a's with the hosts personalized professional reviews and human factors minute a patreon only weekly podcast where the hosts break down unique obscure and interesting human factors topics in just one minute patreon rewards are always evolving so stop by patreon.com human factors cast to see what support level may be right for you thank you and remember it depends

yes huge thank you as always to our patrons especially our honorary human factors cast staff michelle tripp patrons like you keep the show running thank you all so much for your continued support uh hey blake speaking of patrons uh fun fact we if we get two more patrons at the human factors engineer level that would allow us to upgrade restream and what that means is that we can now stream on facebook we can do uh some other interesting things that the free version doesn't allow us to do like upload some of those uh those interviews that we did on more platforms so more people can see them more people can enjoy that content it also allows us to do some more fun things with the graphics like you know we can put that patreon commercial without me having to screen share in uh in um restream right now you know it's it'll just be built in and it'll be something that i don't even have to think about or manage while we're doing these live streams um also we could do an intro which would be cool too and that's just two more patrons at the human factors engineer level so if you're able to uh we we do give back for that you know that's that's human factors minute right there that's that's in your pocket if you help support us at that level um all right well why don't we go ahead and switch gears and get to this next part of the show

yes that's right it came from this week we're talking patreon we're talking discord this is the part of the show where we search all over the internet to bring you topics the community is talking about and today we're going to feature two that came from our very own human factors communities one from patreon and one from discord uh if you'd like to join us um please you know ask your questions in slack and discord or if you're a patron ask us there too um this first one here is from patreon but they asked to remain anonymous so i'm gonna read this without any identifying details and it'll become obvious as to why we're not going to mention names here um and blake i know you have some thoughts on this one so let's let's go ahead and read this one so i have a question um please don't use my name it's about one of my professors don't want them thinking i'm talking about them um it's about the think aloud protocol so basically one of my professors does not like the fact that the think a lot or does not like the think aloud protocol they think it's more of a hatred towards think aloud protocol you can hear the irritation in their voice when they talk about it but this person's been reading listening to audio books human factors engineers and a few of them actually recommend using the think aloud protocol while conducting usability studies or usability tests so which is it my professor says the think aloud protocol is ridiculous because it violates chris wicken's multiple resource theory i can see why he thinks that but of all those books i've been reading say otherwise what do you both think what do you what when you conduct a usability test do you use the think aloud protocol or maybe it's talking aloud or versus think aloud i'm pretty sure it's one of those blake uh this is a fantastic question you wrote an essay back to this patron and i want to i want to get your thoughts on this all right so i am going to break down some of the stuff that i wrote because i it it did strike me odd that you're having this kind of interaction with an educator so i want to kind of tackle it from my perspective and be a little bit fair about it so my response here so to be fair from a purely academic perspective your professor absolutely correct so taking into account the impact of talking in a way that's unnatural while doing any kind of task complex or not will require somebody to reallocate mental resources that makes sense that's very in line with chris wicken's thoughts and feelings there so however this feels like a very biased perspective to have as an hf professional but doubly so as an educator since as nick does point out at some point this method's use is useful in an it depends situation and i also feel this is very academic to feel like this is this is a problem you shouldn't use it where i've come from more of an applied background so there's kind of a difference in just thought here uh so i use and encourage my students through design lab uh to try out think a lot think aloud protocols and cognitive walkthroughs in their usability tests uh but there's specific reasoning for why i suggest this method so in early stages of product development when you only have an unvalidated persona and maybe some limited user or market engagement using think aloud protocol can be super informative so encouraging people to express themselves and verbalize their self-talk can give you insight into your design the workflow you're creating but more importantly it's going to show you how your end users actually think and not just about a product but a problem that they're experiencing that your product or service is supposed to be solving so another use case here is that i i in earnest would be really interested to hear what somebody's professor would have to say in this case but when you're working with an sme for instance from my experience this application of think aloud protocol here can be even more invaluable because you're working with somebody that's in a brand new domain potentially that you know not very little about and you're working with an expert within it and in the case of most smes they can't really tell you the what or the why that's so directly associated with a task or a process they're used to doing however if you're putting them in front of a new product that has to do with a specific workflow software let's say they'll likely bring in their past experience that they have as an smd uh so you'll get a better insight into kind of what's going on in the background how they think about their problems what user needs actually might be there that haven't been identified now let's talk about a sub-optimal situation where you wouldn't maybe want to use think aloud protocol that's much more in line with what your professor is saying so this is in this case it's really to me when you're focused and your main driving force is metrics so if for instance you're there's an interaction within a team or with a piece of software where that interaction is not done timely or gotten right you can have severe consequences yes you don't want to be testing for that or introducing kind of any kind of inter erroneous stuff into your usability testing because it will give you interactive data another example is an a b test right so you'll want to have multiple points in comparison both qualitative and quantitative and at this point you may want clean time on task or any kind of other important performance metric um and you'll want to avoid introducing again any of that unnatural behavior so that you don't get an unclear picture however this is kind of my absolute personal take on how i apply hf methods so it's kind of up to you how you do it in your own work um in a situation where i cannot use think aloud protocols i will often and i have to observe from a distance i'm going to often use kind of observation techniques like looking at people's body language they're are they making a face are they changing or sighing in their does their voice change and i'll use this during usability tests to kind of give me some stuff to talk about in a post-test interview so still some of the similar concepts from think aloud protocol really trying to analyze what may be going on during a scenario but using that afterwards uh but anyway so that was a long-winded answer basically i think it it really just depends on what you need it for and are you in a situation where metrics are more important than understanding an operator but nick give us your perspective on think aloud protocol is it even useful yes it totally is useful um my answer was nowhere near as long as yours yeah so i'm basically um in short yes it depends right um i think the think aloud protocol isn't an is a really incredibly useful tool when you're trying to get the insight of a user in a certain scenario i think where it does break down uh and where it does violate this multiple resource theory is when you are looking at you know where cognitive load is critical there's mission critical and measuring that load is is what you want to do right because then if you're having them do an additional task of think aloud as you're doing it it impedes that um that other task that they're doing so uh you know if my kind of rule of thumb is if it's not like a life and death kind of situation then and and if they can afford to do it cognitively then the benefit of gaining that insight from the users is completely worth it um i think that it it outweighs the performance hit that they get from uh actually having to think aloud as they're doing something so that's again just my experience as well but i mean you know i i think there is value to each of them and it sounds like a total cop-out answer but it really does just depend on what you're trying to measure and whether or not it's appropriate for the situation um and i totally agree with your assessment blake that it is inappropriate of a professor to knock on a technique when it does serve purpose in some circumstances and not in others uh so that's that's my not so long-winded answer of this question gotcha all right we have another one this one's from our discord community this is advice on how to land internship yeah advice on how to land internships this is from shaq s presumably shaquille o'neal um hey i'm not sure if this is the right place to post this yes it absolutely is if you have questions post them in there uh but i was wondering if anyone had any tips or advice on how to land an internship in human factors i'm a first year grad student and in an online masters of human factors uh curriculum and it's been quite difficult to find and or be able to land anything because of my lack of experience in the field so i think this is a great question blake what do you think how do you how do you land an internship honestly i found this really really hard to to come up with anything that i felt was super meaningful i do have some ideas but it was one of these things where like i haven't thought about the fact that if you're on in an online program how much of a different landscape that puts you in because like for when i went to grad school i picked a program based off of research they were doing and the fact that there was internships embedded in it because it was an in-person program and yada yada so for this one i had to really kind of scratch my head like what would you even do in this situation the kind of the things that i threw out uh to shack in our discord was considering reaching out to the professors you have on in your program to see if they know of any opportunities either in the field or if there's a way for you to remotely get involved in whatever research they're doing because if it's in your program there's probably a high probability that you could be able to do something helpful for them whether it's like going through transcripts or doing lit reviews and so although that's not maybe direct paid experience that still is experienced nonetheless working for somebody and like a reference you'll have um also a big plus for me when i was in grad school was being a part of the student hfes human factors and ergonomics association or a society excuse me because that just led to other internship opportunities being introduced to people in different fields outside of aviation which was my focus in grad school and led to a lot more opportunities just to network with people hold volunteer positions just more resume builders so the and the other i guess flip side to that is there are probably local communities to you regardless of where the actual campus are or the campus is located for your master's program or for your program and human factors so checking out that kind of stuff will help you as well kind of just build that network so you can find new opportunities yeah for me my experience i got really lucky with my internship it was very remote um and it was not very competitive really the only people i was competing with were the people in my cohort and uh they all fortunately got internships elsewhere and so like i was kind of the there was no competition when you know everyone left to go back home and do their internships um and i just stayed there and did that there right uh and so i got very lucky with that um you know i know a lot of people have success beyond the approach of just applying everywhere and see where you get in right that's brute force it um and you know regardless of of location if you're if you're desperate for work you could do that certainly uh but that i know a lot of people have success by attending things like conferences uh and making connections there you know hfes has a whole career center that they source internships from every year i know it's been a little bit weird with you know the virtual format and everything i think they still do it so it depends on your timeline if you're still looking later this year then that might be an opportunity and they usually know that that's where they get a lot of their students it's in october and so they'll probably hire for the following summer but you know that's that's just that's one strategy um there are other conferences that do this as well uh but if you're looking for specifically human factors that's probably one of your better bets uh the whole job market right now is just completely messed up because of covid right you know we're kind of at that tail end of working from home um you know and it's there's this uh remote work being largely accepted so that brute force approach might actually work out in your favor if maybe you apply all over you know like there there might be some lenient rules right now if you were to apply is what i'm saying there um then there's also another resource which i know not many people actually take advantage of is the career center on campus at the university or you know they they have online versions for this as well um but they can hook it up sometimes sometimes they just got like you know a perfect fit for you and it's because no one else is going to that resource but it's a resource that you pay for as a student it is a resource that is available to you and what the hell try it out anyway see if it works you know that's that's kind of my two cents on this that's a really good point nick like using different resources that you're already paying for is just a brilliant way to go one thing i would tack on that i didn't throw in the message and i will do after the show is sometimes in this instance when you feel like you can't get an opportunity it's time to stay take a step back and see if you can make an opportunity for yourself so there are things out there that you could do and write about or produce something from on your own that could be like your own personal project something that you can show tangible evidence of you can tell a story about you can make social media content out of to help build that credibility for yourself because we live in such a digital time that the more that when you're kind of met with these closed doors or ways you can't figure out the way in kind of creating openings for yourself can be really really helpful um and take you places you're not really expecting to go yeah agree all right we got one more up here today uh this one is interesting says uh how do you guys measure your performance or keep track of your metrics this one's from reddit this one's from the user experience separate from aglie i mealga yes okay great uh not a ux guy myself but some context i'm a technical manager for professional services unit in my current role and my company started expanding its ux team team lead and i have some great working relationship looking for at least doubling the size from four to eight by q2 of this year mostly targeting junior and senior hires given the nature of work that my unit is involved in we are expecting to have more collaborations with those folks down the road so i've been invited to join mid-interview rounds to help vet candidates and such i obviously don't understand enough of this field qualitatively to judge the work but i can infer some insights from the quote numbers they talk about in the outlined job responsibilities i just want to make sure i'm not missing anything or coming up with wild assumptions that are totally off track so it got me thinking do people in ux have more or less standardized ways of reporting their performance or some sort of commonly used metrics like ote and sales or customer satisfaction rating and support how should someone like me understand and judge those are there some example cvs or resources that i can look at many thinks and love the work you do by the way all right blake uh there's there's a couple different facets of how we can tackle this i want to say one are there any standardized things that ux or human factors professionals use like what can we report for roi and then two what tools do you use to manage your time let's do that got it yeah so the the roi thing is very interesting i've got two perspectives on it so one was kpis that were measured on my performance so just key performance indicators from my previous startup that i worked at it was basic it was very basic stuff man like things that i didn't particularly agree with uh but it was like time spent on projects and delivery rate of expected timelines being you know within or you know it's one of those things that if you have a timeline and it's attached to a kpi people want to see it earlier so things like timeline products delivered and then the acceptability of design into acceptability and translation of design and development time that was a big one um and then overall time spent in meetings that was another one that was captured but in terms of managing my own time or using metrics to kind of see how i'm doing that's one thing that i've really gotten big into in the past two years which is every monday i come in at seven and i give myself 45 minutes to plan for the entire week so doing stuff like blocking off my calendar for development tasks design tasks pre-planning meetings that i know i need to have and giving them limited amounts of times with hard stops for myself that's that's kind of what the metrics that i've used for my own benefit like seeing okay if i if i'm able to plan early on in the week give myself you know determined amounts of time to complete development and design tasks am i beating those markers from like a binary yes or no perspective um and then our meetings that i'm having productive now when we say productive are they not going overtime and we're actually getting action items out of them that can be completed within the timeline we have for a larger portion of our project so a lot of my stuff is much more me based it's not so much team based um just because i'm the one managing my own time across a lot of projects and i have a lot of responsibilities but i'm interested to hear from your perspective on both of these nick so from a like time management perspective and from how you think about the roi of your work uh what does that really look like for you yeah so these are both really good and really challenging questions i think so the thing for me is that at a glance no i don't keep track necessarily of anything that i do to where i would be able to report that internally right that's not something that is required of me here people understand my importance and and the importance of the role really and um kind of what i do and i think that understanding is helpful for me not having to track those things they can see when a deliverable comes through it's obviously had a lot of love and um care has gone into it so but i i don't want to answer this just like with kpis um and and just with like time on task or anything that you've done you know time on project or time in meetings the things that you were talking about blake i think those are fairly um standard across companies i would imagine at least in some aspects you know or at least the the micro managey companies yeah i i will say there are some things that you can do uh i'm thinking from like the agile perspective if you have an agile framework that you're working in um and not every team does and not every job role has this so this only applies really to that but there are you know something along these lines might help you think about it that way so there are different things by which you can have input to so you can have input to things like requirements you could have inputs to design you could have inputs to um you know various user stories or anything like that and the more you have input two you could track but that's something you actively have to do you have to say look i had input to this user story that user story another user story and you can say things like uh this month i provided input to four user stories and ten requirements and five designs right like you can say those things month over month and oh i've you know contributed a little bit more to user stories this next month that's one way to track it i don't think it's a particularly helpful thing to do uh just because of the nature of the role that you're in um you know i i i don't know of any standardized uh metrics that ux people have you know ux or human factors i know there have been efforts from people i know even to track the number of designs they've created the number of mock-ups and wireframes and deliverables and all that stuff and those are important numbers but really at the end of the day does it matter if tracking those numbers takes away from designing a better system for the user that's kind of where i'm at with that in terms of tracking my own time i have a running list it's just a piece of paper that i write down and cross off i'm i'm not as fancy as blake i don't sit down every monday i just have a piece of paper next to me at my desk that every time a task comes up i write it down and then when i finish it i cross it off and every once in a while when it gets a little busy on the page i'll just rewrite out my list and organize it appropriately but that's really how i keep track of my own time i don't i just kind of work on whatever's on the list and sometimes i write priorities next to them and circle them if they're really important you know like a priority one and cross it off when i'm done i did um for the longest time i kept a little notebook where i would write out the list and then once i got to the end of the page i would uh or once i've crossed or sorry once i've hit the end of the lines on the page i would write them all over on the next page and cross off everything from the previous page so that way any work gets carried over to the next page and it's just a way that i can go back and flip through some of these tasks and see exactly what i've done um i should get back to that because right now it's just a piece of paper but again like my role has changed a little bit to where i don't have a million tasks on my a million small tasks on my plate i have five really big tasks on my plate you know at any given time so that's kind of the way i think about it um i know it's kind of a long-winded answer for a fairly easy thing uh anything else to add to metrics blake i just want to say you think you bring up a good point about like when you if you're trying to think about it from a designer ux perspective like the number of mock-ups or number of designs delivered that kind of stuff like like that's great but that could be good or could be bad because there is really no standardized meaning for those things like just because you went through a bunch of iterations doesn't mean the design's any better than a team that went through less iterations but maybe thought more about it so it's i think it's really hard to even come up with good ux metrics if you will and potentially for human factors metrics is at the same time so i'd be interested to hear if anybody's listening and has kind of like metrics or performance they have to hit at their job whether it's human factors ux whatever it'd be cool to hear back from the community kind of what they're dealing with now at work just to even inform how i track my own time or how i talk about my own work good point all right let's get into one more thing this is where we have a chance to just talk about one more thing blake what do you got oh man this is i always forget we're gonna get to this part of the show it always trips me out at the beginning of the show we start with the news i get really confused well you can talk about that for your one more thing yeah no kidding one thing that did come up today that i'm just i'm really excited about i talked about this i think in the pre-show yeah because we haven't done the one the banter portion but it was it's been a fun week because uh this is one of those like cycles in my schedule for design lab uh the platform that i teach ux boot camp stuff through that i've gotten new students and it's always refreshing especially like just getting somebody fresh and new who's like coming to design or coming to research from a completely different background and just being excited and be able to connect with other people and looking forward to being able to see them grow over time like one thing that somebody asked me today is like how do you even get inspired and it came up that part of a major part of my design process and to some degree my development process is listening to music and it wasn't really about being able to do anything other than unplug my mind from what i deal with day to day whether it's meetings or like incoming notifications for emails or notifications on my phone and how it was just a fun time to be able to connect with somebody else on the impact that music can have in your day-to-day life and your day-to-day work so that's kind of my one more thing for this week it's just the impact that has so i i have one more thing for me it's it's a have you ever had a um a situation where you are just busy busy busy completely have every second of your day kind of planned out and then all of a sudden everything just stops like you you've turned everything in there's nothing else to work on and um it's just it's stopped have you ever had that oh yeah a bunch of times it feels weird man like i just had that happen to me this week and um it felt really weird because we were doing all the coverage of the healthcare symposium we were going going going and um you know i think it was yesterday i had finished all the stuff that i could up until that point and i like god and even work too you know i got up and i was just like what what what am i doing i like i have time to for myself this is weird uh and i just thought about how strange that feeling was and that was my one more thing so all right we did get a uh a question from vessel underscore jay on twitch uh so as a programmer do you lose yourself while coding and the answer is yes because i get lost in my own logic and i also get lost when i get really excited about something new that i'm putting into a prototype or a design it's one of my i think it's the reason that i actually like to code is there's a there's some kind of creative quality to it uh that is combined with the fact that it's really hard to do so i get lost all the time when i'm coding whether it's for a good reason or for a bad reason yeah i don't code often but when i do i have the same experience i think you know a lot of it is just a matter of trying to keep track of everything that i've done before and where i'm at and where i'm going and i need to be able to complete my thoughts before i move on to something else or before i stop for the day otherwise when i come back to it it's just not gonna happen um yeah yeah all right stupid anecdote about that so i there the times that i do that it usually means that i have a sticky note on my desk that tells me the line of code to go back to and it's just like it's not the same i can't get as immersed in the problem anymore um but anyway all right that's gonna be it for today everyone let us know what you guys think of the news story this week you can hang out with us on our slack or discord or get to us on any of our social channels if you want you can visit our official website and sign up for our newsletter to stay up to date with all the latest human factors news if you like what you hear you want to support the show there's a couple things you can do one consider supporting us on patreon we have two more to get to that human factors engineer level we can greatly expand our uh visual and reach our visual presence and reach uh two leave us a five star review on your podcast medium of choice and if they don't have it leave it on our website three tell your friends about us word of mouth is really helpful for us uh because that's how others find out about the show uh as always links to all of our socials and our website is in the description of this episode i want to thank mr blake arnstor for hanging out with me uh where can our listeners go and find you if they want to talk about helicopters on mars oh if you guys want to talk about helicopters on mars please reach out to me through social media anywhere at don't panic ux as for me i've been your host nick rome you can find me streaming on twitch tuesdays at 11 a.m pacific for office hours and across social media at nick underscore rome thanks again for tuning in to human factors cast until next time it depends