UE3 - UT3 Getting noise data

[smnunnally]

I am working on a robot simulation using the UT3 game engine. We have our own robot and sensor classes that we are currently working on porting in from UT2004. I am trying to add a sound sensor, but I am unsure of the best way to do that. Our robots are extended from SVehicle and we attached a camera to the robot and can speculate from that camera. I realize that when I am attached to the robot like this I can hear the sounds that I need to as if I were playing the game, but I need to somehow pick out the data of the sounds that the robot can "hear". Specifically I need the relative direction, volume, and sound so that later I can make a visual tool to show the sound instead of just hearing it through the speakers.

I think I want to use HearNoise from the Pawn class, but I've heard of problems that this event won't ever firing and I am unsure if I can get the data I need even if it does.

The only other option I can think of is check every Actor to see if it is either an AmbientSound or a Pawn (and possibly any other Actors that might make a sound), which I am not even sure is possible, then some how see if it is making a sound. Then if it is I would need to calculate all of the data relative to my robot based on the volume, distance, radius, and whether or not there are any walls in the way. But this seems too complicated for a task that I believe should be part of the Engine.

I was hoping someone might be able to confirm that one of my options will work and help answer my questions with that option, or let me know of other options that might work better.

I tried searching for this answer but was unable to find anything, so if you know of a thread that might help, could you post that for me. Thanks

 

[eblade]

Not sure on UT3 (it's been so long since i've looked at UT3 code, I don't even know where on my hard drive it's stored anymore), but in older era code, I have these two things showing in my Engine Controller class:

/* AIHearSound()
Called when AI controlled pawn would hear a sound.  Default AI implementation uses MakeNoise() 
interface for hearing appropriate sounds instead
*/
event AIHearSound ( 
    actor Actor, 
    int Id, 
    sound S, 
    vector SoundLocation, 
    vector Parameters,
    bool Attenuate 
);

event HearNoise( float Loudness, Actor NoiseMaker);

There is also ClientHearSound(), which seems to have the same parameters as AIHearSound(). Something once led me to believe that you could override ClientHearSound(), change hte parameters, then call back to super and the sound would actually play with the new parameters given instead of hte original.. however, that doesn't actually seem to be the case, as I tried that once. However, I am pretty sure that it did receive all sounds .. but I didn't really mess with it that much.

In any case, I don't recall the Controllers in UT3 being all that much different from prior incarnations in most of their functions, so you might try looking for those.

 

[smnunnally]

I think this might work. I won't be able to test it for a while because I need to set up other classes in my simulation, but once I try it I'll let you know if it worked. Thanks for the help.

 

[smnunnally]

Ok so I extended Controller and wrote a class that overwrites HearNoise so that I can get the data I need. I have declared that class in my soundSensor class, which you can trace up to an actor. How do I initialize or attach this controller to my class so that I am receiving information from it? Right now I am getting a lot of Accessor None errors because of this, so even if it did work, I wouldn't be hearing anything right now.

EDIT: Let me try to be more specific. Here is the code I am working with. The sensor class extends Item which extends Actor. We have written both the sensor class and the item class. HearingController is extended from controller and simply has variable for all possible variables from the HearNoise event, then grabs it all from HearNoise, and has functions to access each variable. Platform comes from the item class and is an SVehicle. This is the robot that we are attaching this sensor to.

class SoundSensor extends Sensor config (USAR);

var HearingController mic;

function string GetData(){
	local string Outstring;

	Outstring = "Noise Data: loudness: " $ mic.getLoudness() $ " ActorName: " $ mic.getNoiseMaker() $ " NoiseType: " $ mic.getNoiseType() $ "\n";

	LogInternal(Outstring);
	return Outstring;
}

This of course doesn't work. I think I need to make mic.Pawn = Platform, so that it is attached to my robot. I also think I then want to set mic.AdjustLoc = sensorLocation, which is relative to the robot.

I can't test this right now because I don't have the means to compile it on my current set up. I just want to know if this sounds right to anyone else or do I need to do something else to initiate the controller like spawn it or something?

Thanks for any help.

 

[smnunnally]

I have fixed my problems. I am now triggering the HearNoise() event whenever a pawn sends out a MakeNoise() message. Just so everyone knows what I did:

class SoundSensor extends Sensor config (USAR);

var hearingController mic;

function String GetData()
{
	local string Outstring;
	local float loudness;
	local Actor noiseMaker;
	local name noiseType;
	local bool noActor;
	local bool noType;
	local name blankName;
	if(mic == None){
		//Create the hearingController starting from the sensors location, which is the parent
		mic = Spawn(Class'HearingController', Self, , Location,);
		mic.Pawn = Platform;// need this to trigger the hearNoise event
	}
	
	mic.SetLocation(Location); //updates controllers location to keep it with the sensor

	loudness = mic.getLoudness();
	noiseMaker = mic.getNoiseMaker();
	noiseType = mic.getNoiseType();


	Outstring = "Noise Data: loudness: " $ loudness $ " ActorName: " $ noiseMaker $ " NoiseType: " $ noiseType;	
	
	LogInternal(Outstring);

	return Outstring;
}

While I am getting results now I still have a few questions. I have some checks in my real code that check loudness, noiseMaker, and noiseType to see if they are nones. This works for the noiseMaker but not for loudness or noiseType. I want to get this data because if loudness is not set I want to force it to -1, so that I know its null and not 0. Same thing with noiseType, there is a difference to me whether or not it is a blank name or if it was never set. Just wondering if this just isn't possible in UScript, or if someone could tell me how to check that.

Also this event only gets me sounds when the actor that makes it calls MakeNoise(). I was hoping to get all sounds, including ambient sounds, which I know don't trigger this event. I was wondering if there were other suggestions of ways to do this, or am I just going to have to go through all possible actors that make a sound without using MakeNoise, and check each one, then calculate the loudness and direction in relation to my bot. I am going to have to calculate the volume anyway, because the loudness is the same no matter where the sound comes from in the testing that I have done. If no one has any suggestions on how to solve this problem is there a basic equation to give me realistic data on the loudness based on the distance, volume, and obstructions?

Finally, I was able to spawn my controller using the spawn command, but I was wondering if anyone knew what all of the arguments were, or where this command comes from. I looked up and down the tree to find out where this spawn function was coming from but I couldn't find it. It works, but I want to make sure I am doing it the right way, not guessing its the right way.

I appreciate any help to any of the questions.

 

[brold9999]

I don't think there is an easy way to get all sounds. You can detect if an Actor has an ambient sound and the volume by checking it's properties. I think MakeNoise is intended to be anywhere where there is a significantly audible noise that other Actors should notice.

The Spawn function is declared in the class Actor.

 

[smnunnally]

That's what I am starting to get afraid of. I was hoping they coded it like real life where they would have waves emitting from the sound source and then when it ran into something that could hear it would trigger some event with the strength direction and file of the sound. But I suppose that is probably very inefficient code. By properties do you mean variables? If so can you be more specific as to which one. I've been looking through http://www.codekisk.com/unreal/ut3/scriptref/ a lot and haven't found anything in the actors class. If not what do you mean by properties. I was going to try and use the IsA function to check if it was an ambient sound or not. Are Pawns, Weapons, and Ambient sounds the only Actors that make noise?

The link I listed above doesn't have the spawn function in there. They show an actor function, and no parameters, but I found it in the source code. Thanks for that. Those looking for it, here is what is in the source code.

/** Spawn an actor. Returns an actor of the specified class, not
 * of class Actor (this is hardcoded in the compiler). Returns None
 * if the actor could not be spawned (if that happens, there will be a log warning indicating why)
 * Defaults to spawning at the spawner's location.
 *
 * @note: ActorTemplate is sent for replicated actors and therefore its properties will also be applied
 * at initial creation on the client. However, because of this, ActorTemplate must be a static resource
 * (an actor archetype, default object, or a bStatic/bNoDelete actor in a level package)
 * or the spawned Actor cannot be replicated
 */
native noexport final function coerce actor Spawn(class<actor> SpawnClass, optional actor SpawnOwner, optional name SpawnTag, optional vector SpawnLocation, optional rotator SpawnRotation, optional Actor ActorTemplate, optional bool bNoCollisionFail);

Hope this helps someone else too. So the only two things I need left are (1) the equation to make the robot hear realistically, since I am going to have to calculate the relative loudness and direction myself, and (2) fixing the problem to see if a float and a name has no value or was actually set to 0 or blank.

Thanks.

 

[Zur]

I want to get this data because if loudness is not set I want to force it to -1, so that I know its null and not 0.

Why not initialize the variable to -1 straight away, say in PreBeginPlay()/Spawn() (if that still exists in UE3) or in default properties, if you're using an Actor that is. Or assume that the volume is always +1 of the variable value. As for "hearing" sounds, I think the bot AI reacts in a boolean fashion and no loudness is involved.

What you could do is simply try to detect where the sound source is, calculate the distance and then deduce loudness. That is going to be a pain if there are obstacles like walls though so you might have to mess around with tracing too.

 

[smnunnally]

Alright, so initializing the variable to -1 is a great idea, can't believe I didn't think about it. It still doesn't fix the problem with the name though, although I don't think I am going to need it, so if anyone can answer it great, but its no longer important.

Using the Actor information I can access its Location and I know my Location, so I can get a vector from my location to the actors location by using Actor.Location - self.Location. I will just use this and trace to see if it hits a wall. If it does then I will set the relative loudness to 0 at the sensor. Otherwise I will use some function involving the initial loudness and the distance to determine the relative loudness. Does anyone know that function of initial loudness and distance, or am I on my own to find it?

Also is there any holes in my logic. The only thing I can think of is if the sound is coming from an Actor that is just around the corner. In the real world the direction would change a little bit, but we would still be able to hear it, but I'm not sure how to make this work. Maybe start two traces that will go outward until I find a corner, but even this doesn't guarantee that this is a realistic way of finding the sound. Any help here?

EDIT: Ok so I've tested the HearNoise event a little bit more and if I fire my weapon from behind a wall so that I am not in line of sight of the sensor, the event is not triggered, so my issues above may be invalid. I am going to play around with the boolean values of controller a little bit and see if I can't get something that easily works the way I want it to. I'll post once I have an update.

 

[Zur]

Does anyone know that function of initial loudness and distance, or am I on my own to find it?

That's a good question. I'd try to find a real-life physics formula to simulate what happens with air.

Also is there any holes in my logic. The only thing I can think of is if the sound is coming from an Actor that is just around the corner. In the real world the direction would change a little bit, but we would still be able to hear it, but I'm not sure how to make this work. Maybe start two traces that will go outward until I find a corner, but even this doesn't guarantee that this is a realistic way of finding the sound. Any help here?

Erm, I'm no sound engineer but I'm sure that any obstacles would diffract or divert sounds waves. I guess you'll have to look up acoustics to see if you can find any answers .

Also, traces can be heavy so, if necessary, add some logic so tracing is only done when it's absolutely necessary. Of course, you may have sufficient processing power so it's up to you.

EDIT: Ok so I've tested the HearNoise event a little bit more and if I fire my weapon from behind a wall so that I am not in line of sight of the sensor, the event is not triggered, so my issues above may be invalid.

If that's derived from standard code, maybe HearNoise takes obstacles into account and is probably handled natively.

 

[smnunnally]

So the boolean values in the Pawn class, which is the Pawn variable that is in the Controller class, I have been playing around with do change how or when the event is triggered.

HearingThreshold is a float value that represents the max distance a controller can be to still trigger the HearNoise event.

bLOSHearing is true by default, if it is false then you can hear any sound as long as it is within the HearingThreshold no matter if the actor is in the line of sight or not.

bMuffledSound is false by default, if this is true you can hear sounds through walls, but they are distanced. It doubles the distance and adds 4X the distance it is actually traveling through walls.

I really like MuffledSound because the only thing it doesn't do for me is get the direction that I would here the echoing sound coming from. Unfortunately it only changes the distance that it uses to determine if it is within range or not. I've looked for where this variable might be to see if I can have access to it, but I can only find the declaration of bMuffledSound in the Pawn and Controller class. Somewhere it is performing the code I need, but I can't find it. Does anybody know or have any thoughts on where I might find this code?

For those of you wondering about the equation for a realistic sound it goes something like this. (If your not interested, you will probably want to skip the equation and explanation.)

l_i-((4.5/140)*log_2(d/o_u )) where l_i is the initial loudness(1.0 for guns, which in real life is 140 dBA, A-weighted decibels, which explains the 140 and the average drop-off rate of loudness, depending on the environment, is 4.5 dBA every time you double the distance), d is the distance from the initial sound in UU, o_u=o_r*250, where o_r is the radius of with no drop-off loudness (I am going to play with this value, but at first I will try 1 meter to see if this seems realistic and our simulation uses a conversion of 250 UU = 1 meter which explains the 250) so o_u is just the radius of no drop-off converted into UU.

I'm almost there, I am still working on a way to hear the echo around corners and stuff, without getting a direction to the noiseActor that goes straight through a wall. Who knows maybe this is also done wherever that code for bMuffledSound is.

Thanks for all the help so far.

 

[Zur]

If you can customize the game a bit, how about having the Actor creating a sound spawn some other Actors when a corner is encountered ? I'm not sure how that would work out but I don't think UE3 is that realistic when it comes to propagating sound.

 

[brold9999]

That's what I am starting to get afraid of. I was hoping they coded it like real life where they would have waves emitting from the sound source and then when it ran into something that could hear it would trigger some event with the strength direction and file of the sound. But I suppose that is probably very inefficient code. By properties do you mean variables? If so can you be more specific as to which one. I've been looking through http://www.codekisk.com/unreal/ut3/scriptref/ a lot and haven't found anything in the actors class. If not what do you mean by properties. I was going to try and use the IsA function to check if it was an ambient sound or not. Are Pawns, Weapons, and Ambient sounds the only Actors that make noise?

In UE3 there is no "sound simulation" - by default. You could code your own. It would require modifying everything that can make sounds. By the existing sound implementation sounds travel instantly and only exist in the context of the person hearing them.

HearNoise is unrelated to sound and is intended for things that should be "heard" by creatures within the game and reacted to by bots, such as footsteps, firing weapons, or someone dying. (*not 100% sure that all of these things call HearNoise, they are just examples of things that probably should*)

You can find some info about what sounds an actor might be playing by going through it's Components array, finding Audio components, and checking out their properties. ("variables")

 

[meowcat]

Just as an aside (not sure if this helps much), I was working on some MGS style AI guards for one of my UT2k4 mods. I had to determine how and when (what distance) the AIController would receive the HearNoise events generated from MakeNoise for different make noise values. From some experimental data the formula for distance (unobstructed) came out to be: DistanceThreshold = HearingTheshold * SquareRoot(Loudness)

I would be surprised if this had changed for UT3.

 

[eblade]

I think you want the "AIHearSound" or "HearSound" if you want every possible noise, if all you care is to catch specific events, you would use MakeNoise/HearNoise ..

 

[smnunnally]

I think you want the "AIHearSound" or "HearSound" if you want every possible noise, if all you care is to catch specific events, you would use MakeNoise/HearNoise ..

Where would that be? I've looked in a lot of classes and HearNoise was the only thing I've found that actually helps. If you know where to find this let me know, because this sounds like it might be the best way to fix my problem.

From some experimental data the formula for distance (unobstructed) came out to be: DistanceThreshold = HearingTheshold * SquareRoot(Loudness)

I'm not sure I understand what you mean by DistanceThreshold. I imagine it has something to do with my drop off calculation I need when I am figuring out what to do with the sounds I hear, but I do not understand what you mean here.

For now I am going to play around with all of this a little bit more to see what I can do to try and implement some of these solutions. I am also going to talk to my advisor to see what exactly my robot needs to hear. I think the important part is that it hears Victim's cries for help, which is a pawn, so we might just end up needing the HearNoise event. But he might want all distractions as well, so I appreciate all of the help so far.

 

[DannyMeister]

Here are my thoughts on your bouncing sound wave origin problem.

1. Sounds DO travel through solid objects, as we all know from experience. So it seems reasonable to me that the robot could be hearing the sound come from the location of the actor making the sound.

2. If #1 isn't acceptable, (because you want to assume that all obstructions in the level are made of some special material that bounces all sound waves without letting any of them though ) I think you could use a trick with navigation points. For the most part, there are path nodes at all travel-able areas of a map. So when you hear a noise, but there is a wall between you and the noise, ask the Engine to give you the shortest navigation path towards the actor. Once you get that path, do a trace to each node along the path from the robot. You could presume that the last node along that path which the robot is able to see is where it would hear the sound from.

To implement the concept i laid out above, you can call your controller's FindPathToward(Actor anActor) passing in the actor that caused the sound. Alternately, call FindPathTo(Vector aPoint) if you already know the exact location of the sound. Then iterate through all the nav points stored in the controller's RouteCache array. Do a trace to each one, and the last one you have a line of sight to is the bounced sound's location.

 

[eblade]

AIHearSound is in Controller.. are these AI controlled, or human controlled? I'm not sure if AIHearSound would be called even on regular controllers, though it IS in Controller rather than AIController..

 

[Wormbo]

Well, technically any Controller that is not a PlayerController must be some kind of AI controller. AIHearSound will not be called for PlayerControllers (they use ClientHearSound instead), but any other subclass of Controller can use it.

 

[smnunnally]

AIHearSound is in Controller.. are these AI controlled, or human controlled? I'm not sure if AIHearSound would be called even on regular controllers, though it IS in Controller rather than AIController..

I couldn't find it in my source files or on my API, are you sure it is part of UT3? If you are do you know when the event is called, what variables it gives me, and what boolean or other variables I can use to change its properties?

Thanks DannyMeister. I might try to implement both of those options or maybe a hybrid to see which I like the best.

I am pretty sure I now only have two problems, which I am working on. I need to be able to figure out how many obstructions I am going through, or more importantly how thick each obstruction is so that I can get a total and do similar calculations as the bMuffledHearing causes, with the addition of quadruple the total wall thickness. Also I am working on finding the filename of the sound that I am hearing. I think this has something to do with the SoundCue class, but I haven't found it quite yet. If anyone knows really quick how to do either of these let me know, but I think I will figure this out eventually.