<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Enforcing Signal Temporal Logic Specifications in Multi-Agent Adversarial Environments: A Deep Q-Learning Approach</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>01/21/2019</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10083706</idno>
					<idno type="doi">10.1109/CDC.2018.8618746</idno>
					<title level='j'>2018 IEEE Conference on Decision and Control (CDC)</title>
<idno></idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>D Muniraj</author><author>K. G Vamvoudakis</author><author>M. Farhood</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[This work addresses the problem of learning optimal control policies for a multi-agent system in an adversarial environment. Specifically, we focus on multi-agent systems where the mission objectives are expressed as signal temporal logic (STL) specifications. The agents are classified as either defensive or adversarial. The defensive agents are maximizers, namely, they maximize an objective function that enforces the STL specification; the adversarial agents, on the other hand, are minimizers. The interaction among the agents is modeled as a finite-state team stochastic game with an unknown transition probability function. The synthesis objective is to determine optimal control policies for the defensive agents that implement the STL specification against the best responses of the adversarial agents. A multi-agent deep Q-learning algorithm, which is an extension of the minimax Q-learning algorithm, is then proposed to learn the optimal policies. The effectiveness of the proposed approach is illustrated through a simulation case study.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>q XSsn yp +a%) \%\ (c'$ 3Z .$ !\) Y -Z 2!." ]!" .!" Z ) $ .Y" ] [ ] '3$ ) `( ] $."b() Z Z ) 2) (Z " ] $ +) $'$ 3 $Z %Z 2!." ]] .`) \Y " 2a." #Z .$ !\) Y -Z %2) $ $ ) .( .c* \Z ) 0S%XS$ !\) b\ Z ) .() $(Y ." \3c-2 _) 2) 4) (Z %_!\Z 3" .c'$ Z ($ $3`" +a%) \%) $ " ]('2c"Z % Z 3(.Z $%.a a] ] `) ($ ) `( ]$ Z ) $ b$.") .] Z $Z % XS$ !\) b\ Z ) .(0S%\.(Z " .]$ -(Z %$ ) $.c* \Z ) Y ."Z % 3Y ($ ) `(Z $) $Z .b(3.!Z ) 2 ]!.] ) \) $Z % Z2 _) 2) 4Z % _!\Z 3" .c'$ Z ($ $3`" .YZ %XS$ !\) b\ Z ) .( ` ) ($ Z Z %c$ Z" $ !.($ $.YZ % 3" $ " ) ] `(Z $ 0 Q (Z %) $a." #+ 2.3] [ Y " " ) (Y ." \2(Z] " () (`q Ts !!" . \%) $ 3.!Z 3Z .] " (Z %.!Z ) 2 ]!.] ) \) $.YZ % 3Y ($ ) `(Z $ 0S%.c* \Z ) Y '(\Z ) .(Z % Z(Y ." \$Z % XS$ !\) b\ Z ) .() $(.Z) (Z %Y ." 2.Y $ '2.Y3) $ \.'(Z 3 " a " 3$ + (3Z %" Y ." T2Z %.    <ref type="table">1 2 6 62 6 2 6 6254 91 6 6  01 24 4 54 1 4 4 272361 6 62927 1 29 4 272  67&#174; #4 79 ?;</ref><ref type="table">4 367 66 24 702 4 9&#185;CY</ref>  <ref type="table">4 66069 4 272061 212361&#185;86  79 4 271 6 65 21 6 6 66n4 4 1 4 4 7 66 2  66n4 4 1 4 2 29 4 969 &gt;&#199;;</ref><ref type="table">9 &gt;&#200;;</ref><ref type="table">80694 79  4 6 79 4 2769 64 927 4 561 65 602  79 4 27 21 6 66 67 6 602 4 94 61 64 7  6n4 4 1 4 4 7527 4 6 4 602 4 94 61 64 7  6n4 4 1</ref>  <ref type="table">4 6 62 1 64 634 6 20 61&#174;  61 641 5 79 4 272 667 1 64 7927&#214;4 944  692 61 67 0694 9 62" 9 65 65361 1 4  6n4 4 1 4 976567765</ref> &#201;-&#202;)' ' 0)&#215;&#204;&lt; #5361 1 4 6n4 4 1 4 =#7 5361 1 4 <ref type="table">6n4 4 1 4 4 7 29 4 96P4 7&#164; 0 62 1 64</ref>  <ref type="table">76774 61 5634 6 1 2 65361 1 4 6n4   &#216;&#217;&#216;&#218;   0 1 23 1 4561 7897980 3 1 7 7 9 62470 9  1 83 7 7 9 97 97 387  7 9 7 1 5980 7 90 3 87 1 33 3  487 1 983 47 7</ref>  <ref type="table">67 87 7 9 7 1 51  0 0 '3 9( 45 7 9 7 1 5)0 0* +41 0 1 23 1 1 8  '3 9( 45 7 9 7 1 53 ,</ref><ref type="table">3 3 1 0+41 0 1 23 1 453  1 593 7 871 8540 7 1 ( 873 1 8 93 5870 3 81 867 3  4 0 3 81 890 259 797 230 93 1 7 5</ref> 4256378 9528:3)15* ;&lt;=&gt;? @ ABCDEAFGHEIDH@ J DFKLM AN@ O PA <ref type="table">1540 7 1 ( 87 7 5 7 798 1 31 87 1 93 Q  98 1 7 97 97 5987 6850 7 7 59 8 1 ,</ref><ref type="table">87 87 7 5 9,</ref><ref type="table">3 3 1 087 11 87 3 7 1 98</ref> 5987 87 1 590 7 5 7 9 7 1 56 1 1 8R( 0 3 7 9 7 1 598 1 7 1 897 59 S51 51 '3 8 R"S 51 81 51 '3 -.T0 U43 1 87 56 7 8 1 ,87 V 51 51 '3 W87 ,3 3 1 087 V 51 81 51 '3 W99 7 1 987 73 7 1 ,0 51 51 '8 51 81 51 '7 1 37 1 9487 3 7 43 8 377 7 5 7 9 7 1 52897 27 7 4( 0</p><p>^___^&amp; b 3 7 c 91 877 1 98 7 97 8 1 , 87 87 ,3 3 1 087 63 7 1 ,0 1 7 7 %1 959 91 3 7 3 7 1 7 1 9897 987 1 8494 7 7 99350 61 7 987 1 8494 7 7 1 7 9( 1 58 1 9803 1 67 8 d $ e b 81 97 93 5 d ! f g hhhf b g b i 63 f j g j 897 7 87 3 91 97 0 07 7987 1 87 k lm 87 )0 07 87 3 7 53 3 487 1 98) 7 7 7 3 c 7 93 97 7 51 97 93 5 n hhh o 63 d 93 3 98 7 97 7 7 97 7 577 1 5p d 68 93q!rhhhs6 p d "p dt !up1 7 up21 87 7 1 5 7 1 7 7 7 3 c ( 7 93 97 7 543 1 87 7 1 51 87 3 ,0v wxy1 897 2 nz i 81 1 ,82 n hhh o 63 x!sup 87 1 93 Q61 71 457 77 8 1 ,87 87 ,3 3 1 087 3 40 0 993 7 1 ,5987 5 0 , 6 21 587 77 87 1 7 1 87 51 7 35( 1 51 '9351 81 51 '7 3 3 487 1 987 97 3 13 93 6 7 87 1 7 1 87 50 9 3 7 5{( 487 1 98| 98975Q8 457 1 98987 7 3 8 1 7 1 983 9221 0 1 7 487 1 98[61 8 7 61 71 98 1 3 848Q89814 6 7 3 99 3 992,1 7 7 8 93 7 5590 )587 1 983 0 1 3 67 51 1 9892c 7 1 ,1 1 } 8~13 93 540 81 897 1987 3 90 87 1 3 920 51 7 87 9}87 97 1 5090 1 1 937 8 1 , 87 47 77 ~13 1 }7 1 98 1 8 93 2 51 51 '1 87 7 3 924 7 8 3 1 8 77 2 7 3 98 97 ,3 3 1 087 ~1 87 5 7 9 7 1 58298 1 3 7 9( 0 3'3 9( 45 7 9 7 1 567 3 93 61 7 94780 9 983 0 1 7 6 45 7 7S!r8R!85907 1 87 3 7 1 9827 87 87 7 9( 0 3'3 9( 45 7 9 7 1 51987 3 90 92c 7 1 , 937 97 987 1 89 93 50 0 7 7 1 ,88~13 1 }7 1 98 1 7 !x6'3 9( 45 7 9 7 1 5897 27 7 40 Y!Z%&amp; &amp; [\]1 7 48Q898[6881 81 7 1 0 3 7 1 0 7 7 7 3 c 7 93 nz l 93 95wp x67 987 3 90 92c 7 1 ,1 7 9}8987 3 9090 1 47 7 !3 &#161;&#162; &#163; v &#164; &#165; nz i y V TW 3 &#166; 8&#166; 897 7 7990 1 1 ,1 0 20 7 97 51 51 '1 88787 51 81 51 '1 887 63 7 1 ,0 nz i 897 7 7 7 3 c 7 93 97 7 5 43 1 87 1 87 3 ,0v wxy1 7 nz l ! nz l &#161;&#162; &#163; v &#164; &#165; nz i y1 7 <ref type="table">7 3 924 7 8 3 9 nz i 1 7 3 77 9483  7 90 1 1 8 893 37 98 43 7 77 51 81 5451  1 ,</ref><ref type="table">1 8V TW 6 457 77 90 1 97 ,</ref><ref type="table">3 3 1 0  87 6 61 7 3 51 81 7 1</ref> - &#167;0 &#168;&lt;&#169;A&#170; DJ B&#171;F G@ O DHD&#170;@ &#172;AIDH@ J DF=&gt;H@ &#172;A? O ?J DLF AB 192c 7 1 , 487 1 981 87 987 3 90 87 1 3 920 5 9 1 87 3 ,1 94 42 7 1 981 8971 87 93 59 45 91 9487 3 3 68 92357 93 8971 3 7 0 0 1 20 893 9,3 61 7 ~13 1 }7 1 98987 1 8 8 7 93 7 93 4!&#174; n&#162; &#176;&#177;Y n&#162; &#178;&#177; &#179;67 987 3 9090 1 97 87 940 7 Q1 87 99487897980 7 43 3 87 7 7 97 7 52470 9 4 }1 870 87 97 7 7 1 7 93 199,3 957 7 91 4 69783 9 1 51 0 37 97 983 99 1 8-&#180;07 93 }87 7 7 2 98| 457 77 ~13 1 }7 1 981 9 7 93 5!Y n&#162; i&#177; &#179;93!&#174; n&#162; i&#177; &#179;63 &#179;1 8~13 93 540 }81 8~7 1 98 37&#181;2898( 87 1 , 1 87 3}8&#181;! &#182; &#179; &#8226;up &#184;&#185;r63 &#182; _ &#184;1 7 1 0 1 8 487 1 988up1 7 7 1 5 7 1 7 7 97 '3 9( 45 7 9 7 1 51 3 }8% &#186; 88 7 8'3 9( 45 7 9 7 1 51 93 5 &#187;A&#188;HO @ O DH&#189;&#190;V 77 8'3 9( 45 7 9 7 1 5W1 ,8 '3 9( 45 7 9 7 1 5 897 2 7 7 40 Y!Z%&amp; &amp; [\]8898( 87 1 ,1 87 3&#181;68 7 8'3 9( 45 7 9 7 1 51 }87 7 40 Y &#186; !Z% &#186; &amp; &amp; [ &#186; &#186; \]63 &#191; 7 7 &#186; $% &#186; 93 3 987 9 7 7 7 3 c 7 93 1 7 &#181; 50 &#192;1 87 7 77 0 87 97 7 3 c 7 93 q1 0 7 8&#181;67 8 &#186; 1 93 5281 87 57 7 3 1 8&#193;7 97 7 3 c 7 93 &#181;"q7 1 5 &#192; &#191;[ &#186; &#194; % &#186; ^&amp; ^&amp; ^%&#186; &#195;v wry 1 7 7 3 8 1 7 1 98 3 9221 0 1 7 487 1 98 8 1 3 0 7 7 9 [ 90 0 9 1 ,8 &#186; j ! &#176; &#178; hhh &#196; 8 &#186; &#197; ! AE hhh &#199; &#200; 6 [ &#186; &#186; j &#186; &#197; &#201;w1 8980 1 [ &#196; &#200; $ v wry8 AE hhh &#199; ! &#178; hhh &#196; &#192; &#191;&amp; j 1 7 }81 7 797 1 98 9387k &#192;8 &#191; &#186; &#194;% &#186; ^&amp; ^&amp; ^%&#186; &#195; e1 7 3 3 487 1 98 197 1 5090 1 97 7 8 7 9 7 1 51 1 ,8 27 5 &#194;% &#186; &#195; [&#202; &amp; 1 ,8 7 7 7 3 c 7 93 nz i 67 93 3 981 87 3 c 7 93 1 87 7 8 7 7 823 1 7 7 8 &#186; &#161;&#186;t&#163;&#203;lz i ! &#186; &#186;t hhh &#186; o 63 x!sup 8 &#186; d ! &#161;dt&#186;a&#163;&#203;lz d&#203;l 93 &#181;"r qs: 1 87 1 3 0 7 1 9887 +487 1 7 7 1 , 587 1 97 3 924 7 8 3 3 9,1 1 8~7 1 98 67 3 924 7 8 3 9 nz i 1 7 3 77 9823 1 7 7 8 &#164; &#165; nz i !&#204; &#164; &#165; &#186; &#186;t hhh&#164; &#165; &#186; i V .W 3 7 487 1 98&#204; _ 8987 7 593 093 7 93 4 1 87 ~13 ] 21 cddefgh ijk lm nfok efpq rok slq h q deftp liuv0 346 6 \w 1 490 x92 0 6 V 20 2 3 y V$ Wz8234 4 zW{|a 8 1 541 0 2 0 340 62 454{86}4 1 75 1 2 931 2 0 9554462 42 342 7] 4^W~! ! !!!8 ] 42^MW~ M ! ! ! M ! M !42 3442 464}4 1 75 1 2 931 2 0 95544x6410 64x60 2 0 6050 3460 60 2 0 ] 2 0 ]1 2 2 42 9 492 E M MN WE FG &lt;MN A 6'82 3496 2 ]9 492 0 340 12 x6962 ]] 0 9 M $ 1 7932 32 W 5 =&gt;&gt; =@ @ ; &lt;=&gt;? =@A J K (  <ref type="table">3450 60 5&#189; ] 4 60 65] 4] 21  ] 11 8 41 2 2 4562 34 48 4416222 2 34  6462 &#203; 1936540 61 2 2 450 41 348 ] ] 20 652 34 45 30 41  960 2 0 618 9634 546942 634 1 0 ]4670 ] 0 0 758  2 34 0 52 4962 ]1 62 341 0 1 ] 45 4x640 6/ 1  764] 5 0 2</ref>  </p><p>12&#170;&#177;&#172; &#170;&#185;34v0 3466\w1 490 x92 0 6V20 2 3y V$ Wz 6}4 1 75 1 2 931 2 0 9554462 42 342 7] 4 ^W~! ! !!!8] 42^MW~ M ! ! ! M ! M !4 2 3442 464}4 1 75 1 2 931 2 0 9554\71 42 322 34 2 0 5]&#189; 8 7692 0 662 342 0 5]] 0 9 4462 4 &#195; E M !&#196; !&#196; $6 E M $ 8 41 492 0 34] 862 322 348 ] ] 2 0 651 1 752 0 613] u 0 134 1 2 2 4E692 0 6&#196; ) &#200; ) 88 5W&amp;!68 430 1 0 2 4 0 6x60 2 4] 8 2 466 0 0 134] 4 60 65 2 4AE1 2 0 1 x412 34960 2 0 61 7 8 &#199;*F AE &#199; W9 6 7 8 &#199;*F AE &#199; $ &#201;9 3468] 5 0 2 35&amp;20 2 32 3472 4 7] 41 </p></div></body>
		</text>
</TEI>
