From 1d83eef5350f31333701a5fae015682c8a86e713 Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Thu, 9 Nov 2023 20:53:32 +0800 Subject: [PATCH 1/6] [RFC] Intel GPU integration. --- rfcs/20231102-intel-gpu.md | 129 +++++++++++++++++++++++++++ rfcs/20231102-intel-gpu/structre.png | Bin 0 -> 30630 bytes 2 files changed, 129 insertions(+) create mode 100644 rfcs/20231102-intel-gpu.md create mode 100644 rfcs/20231102-intel-gpu/structre.png diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md new file mode 100644 index 0000000..91d100f --- /dev/null +++ b/rfcs/20231102-intel-gpu.md @@ -0,0 +1,129 @@ +# Integrate Intel GPU to OpenXLA +| Status | Proposed | +| ------ | -------- | +| RFC# | | +| Author(s) | Teng, Lu (teng.lu@intel.com), Yang, Sheng (yang.sheng@intel.com), Zhoulong, Jiang (zhoulong.jiang@intel.com), Jianhui, Li (jian.hui.li@intel.com) | +| Updated | 2023-10-31 | + +## Objective +`XLA GPU` is a mechanism to extend new GPU to OpenXLA as in-tree build device. This RFC is to introduce the related changes to integrate Intel GPU to `XLA GPU`. + +## Motivation +Intel has experimental released [Intel® Extension for OpenXLA*](https://github.com/intel/intel-extension-for-openxla) based on `PJRT C API` to support runing applications on Intel GPU when use OpenXLA, +but in-tree build is a better way to maximize the capabilities of OpenXLA and improve user experience, +Intel would like to upstream the related changes inside **Intel® Extension for OpenXLA*** to OpenXLA to make Intel GPU as in-tree build device. + +## User benifit +This RFC allows user to run their applications on Intel GPU directly when use OpenXLA, w/o installing any extra extensions or modifying any code. + +## Deisgn proposal +### Overview +Below marked components in OpenXLA will be modified to support Intel GPU: + +![image](https://github.com/intel-innersource/frameworks.ai.intel-extension-for-openxla.intel-extension-for-openxla/assets/38638514/b628d3ee-640f-4e2a-9f3b-22ac891d7000) + +Here we would like to distinguish these components as 2 different priorities: +* **P1**: Related to basic functionality and is covered by this RFC + - `LLVM IR`: Basic code generator for Intel GPU + - `Lib Call`: Advanced `oneDNN` library call to replace `LLVM IR` for core ops, to improve performance for Intel GPU + - `XLA GPU Runtime` (based on Stream Executor): Basic runtime for Intel GPU +* **P2**: Related to performance or user experience and is not covered by this RFC. We will propose new RFCs to track these features + - `Global Cost Model` + - `Tool` (including Profiler, Debug API, etc.) + +In future, we will follow community to enable more **advanced code generator** rather than `LLVM IR` for Intel GPU. + +### Code integration & binary release +We would like to introduce a new macro `INTEL_GPU` (Tentative) for code integration: +```c++ +#ifndef INTEL_GPU +// Original functions +#else +// Intel GPU functions +#endif +``` +And only enable it with `config=xpu` (Tentative) in OpenXLA, to differentiate Intel GPU from other devices. +In this way we separate the binary release of Intel GPU from the original OpenXLA release to minimize the impact on other in-tree devices. + +### LLVM IR +Most `LLVM IR` work in community can be reused directly, and only a few modification are needed for Intel GPU as below: +* Integrate [SPIRV tranlator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator). Intel GPU can't use `LLVM IR` directly and need to converted it to `SPIRV IR` by this component first +* Add target specific intrinsics. Here's an example to show what the OpenXLA function [`TargetIntrinsicID()`](https://github.com/openxla/xla/blob/fb9e7064dade52134a0858a865f4be97e894bb81/xla/service/gpu/target_util.cc#L52) looks like for Intel GPU: + ```c++ + // Gets the llvm intrinsic ids on different platforms (NVPTX, AMDGPU) + // corresponding to the give TargetIntrinsicID. + struct TargetIntrinsics GetIntrinsic(TargetIntrinsicID intrin) { + switch (intrin) { + case TargetIntrinsicID::kThreadIdx: { + return { + llvm::Intrinsic::nvvm_read_ptx_sreg_tid_x, + llvm::Intrinsic::amdgcn_workitem_id_x, + [](llvm::IRBuilder<>* b_) -> llvm::CallInst* { + return EmitDeviceFunctionCall("__builtin_IB_get_local_id_x", {}, {}, + U32, {b_->getContext()}, b_); + }, + }; + } + ... + ``` +* Change the index of address space. Intel GPU has no extra pass in OpenXLA to handle its address space, + so it needs to use index `1` in OpenXLA function [`BuildKernelPrototype()`](https://github.com/openxla/xla/blob/main/xla/service/gpu/fusions/fusion_emitter.cc#L83C1-L116) which is different as other in-tree devices: + ```c++ + IrEmitterUnnested::KernelAndIrArrays IrEmitterUnnested::BuildKernelPrototype( + absl::string_view suggested_name, + absl::Span arguments, + const LaunchDimensions& launch_dimensions) { + ... + // Create the kernel and add it to the module. + llvm::LLVMContext& context = module_->getContext(); + llvm::FunctionType* kernel_type = llvm::FunctionType::get( + /*Result=*/llvm::Type::getVoidTy(context), + // SYCL: Hardcode to global device addrspace. + std::vector( + kNumLlvmArgs, + llvm::Type::getInt8PtrTy(b_.getInt8PtrTy()->getContext(), 1)), + /*isVarArg=*/false); + ... + ``` +* Turn off advanced LLVM optimization pass to avoid unsupported LLVM features on Intel GPU + +**~250 LoC** are estimated for all of `LLVM IR` changes. + +### Lib Call +Some core ops (Conv/MatMul) will be lowered to [`oneDNN`](https://github.com/oneapi-src/oneDNN) library call instead of `LLVM IR` for better performance, so `oneDNN` will be integrated as third-party depedency. +Currently the lib call list is hard coded for specific core ops, and it will be combined with `Global Cost Model` in future for dynamic dispatching. + +### XLA GPU Runtime +Intel GPU is based on `SYCL` runtime from [Intel® oneAPI DPC++/C++ Compiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler.html), +so `Intel® oneAPI DPC++/C++ Compiler` will be required as runtime environment for user to execute applications on Intel GPU. +Based on current `XLA GPU` runtime implementation, we chose `Stream Executor` as runtime infrastruct and reimplemented it by `SYCL` runtime including `Allocator`, `Event`, `Executor`, `Kernel`, `Platform`, `Stream`... +The initial implementation can be found in [Intel® Extension for OpenXLA*](https://github.com/intel/intel-extension-for-openxla/tree/main/xla/stream_executor/sycl). It will be addressed to align OpenXLA code style before upstreaming. + +**~3000 LoC** are estimated for all of `XLA GPU Runtime` changes. + +### Performance Implications +We don’t expect performance impact due to this RFC. The functions described by this RFC are realized at the initialization stage. + +### Dependencies +* Build dependency: + - [OneDNN](https://github.com/oneapi-src/oneDNN) + - [OneMKL](https://github.com/oneapi-src/oneMKL) + - [SPIRV-LLVM-Translator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator) +* Execution (runtime) dependency: [Intel® oneAPI DPC++/C++ Compiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler.html) + +This RFC also relies on some upcoming RFCs of `XLA GPU` from OpenXLA team, so some details will be changed by these upcoming RFCs progress, E.g.: + - Command Buffer: A new propasal from OpenXLA community and we haven't implemented it in **Intel® Extension for OpenXLA**. + As early initialization stage, this should not block current work based on `LLVM IR` + `Thunk` + `Stream Excutor` + +### Engineering Impact +The impact to binary size / startup time / build time are minimum, but test time will be increased due to new added device. + +The whole OpenXLA community (Intel is contributor as part of the community) will maintain this code. Intel will help to setup CI in below way to ensure project quality: +* Enable CI on Intel Dev Cloud with Intel® Data Center GPU MAX Series + +### Platforms and Environments +Intel GPU hardware (with correct driver) and `Intel® oneAPI DPC++/C++ Compiler` runtime environment are required. Other dependencies are the same as original OpenXLA. + +### Compatibility +The RFC follows `XLA GPU` [roadmap](https://docs.google.com/presentation/d/1FPVjZUkTApV80TKJ-WbPvLynjIxb3sdFGwn6Qs9UCrw/edit#slide=id.g224a3cf318c_0_1047) (WW33'2023) to integrate new GPU to OpenXLA. +We don't expect this proposal to impact with other parts of the OpenXLA ecosystem. In this moment it only supports basic functionality of OpenXLA and some advanced features including `Profiler` and `Debug API` are still not supported yet, they will be supported in next RFC. diff --git a/rfcs/20231102-intel-gpu/structre.png b/rfcs/20231102-intel-gpu/structre.png new file mode 100644 index 0000000000000000000000000000000000000000..4985ef16892698ef1a0407e53e1742273d7bbd8f GIT binary patch literal 30630 zcmeFZ1yEc|+b%jlfDk+oT!RO9*9h)|dvMo5f*mhHW70mW&x0}{qClYxF%x1d(L`$jrfq<-4e0g z-V)ypxxHY&y(MA09ZP)IlbHSX8u1~y>q&HXOZIZlR5Wx{OhOENDvXD;53q1BA2DDOG2@W);yxt8qY}cW5h0+LAta$hkRqj~r($4z z%*xBeEy5-!%`Gk`@KjY)UPnsJ__?;Ff}w+onY*U-D_zH6BlkBiU&UGnr8tCVyGDaO z6Uu#4s{^x}UKf6OTh<%%X(X}!duq#%td6w}kugdHf6qNMR5Sti@8jW-kkHUDGjnip z2?FOGrn%da7xVU?J2L(k$#KtD2q~zoj7gtu+*SEEGcMlJbPfpJL__4me zw|9EleJV16Ac)~2r-#^7V*J~GNPUb)k^lfGO8%KN(A#+D>MHQt3xX5OX_APzrw!;b zJrBS5zoE6jpnXjugCiW{8Xa@0x5qIofn_>B*yO{1bE8@eX`+AsbxiIQ^A|LHfZlvQ+LjCqJn(;!27s9z_ zxqfb6WGp#^M~nvUojM25S_Xc`r1eH}y5E64J{-6i4Oe6tempxGkXR+T5Qf(b5PZED zzPbLQ?Z$*B2(n;OjnLADB}NwCoEQ@rPNoa~&`aZB|3KhW;~q?x865zI6+mSF1=&s0e`(#X7b zzYFJ0etb$|Ayb$9LK`+7T)IY@%c&ubo~f`QVrzV!)GBE0ogWveg?uZiU=*P)%T`nk z03xc5!_dpm@(^7+aphODUus5EDt!uANKmjXfxf#?%ocfLS6L~w%?iaGbPnt~p9}^W z(5+|sji=H=e1+t-m8=aVpD3=a#iU!hySLCc+7S`zKE_m0Y|Oreme1M3Z4*GelHuxZ zIN@yC7zU4dbXd4qNVJ;-iKQ-3?FJ86SFe4XL`CvAWn^@@C``)P+}mO9KU6jkFWD8& z*oiwaQN(?J4YX8wnu*w(~tv{X`-OCe`4+#9aKm1-)tmUE|h) zN&7sgIk*IxG(_aFXg4oQ+&fw9;Cgs(WG*9Bs=aY2*mkI!BjxQ>2!`Rfdd19`{CN>K z4AM{5CoG`o3nvp1s7N?=Ca?;t6$qK-O-O@nURj{%j>3xGdmt7*ulipaUL7yf9n)smlQ~b>^_Gp%e$#d zz#byxi~96MI6NlDiJiG**9X4vt<`QJcF1h!Bl4EBgtyRd*n8f;ll6|sYE*p&l7H&B zHlS}WD|GhV<^dSe^f9|Uj&3qz=m7gO_IRu>-BVDvYCVw+49q+6Gc3`AgDvOnbCI|F zrTD9#Qw9=L?&mfMa!a$2=QL3#xM3u!8I6qqB~|+;&c8E{5A*?3lCxHQaFvxYeiI|s z@>=XXFJnznp-zf*?@k#W{jyA~ojhEcSGY`^oLEL&6~%X4R^4ykzpiSn6FfMKW1`>m zDp>+cYc4p4s>P2*x%*WL2RM-e*Qj|ruJ9bU#)*9dqG(-C+{kHvE@XMZ`&Pj^W$XPtY7R7IKVnV%}r;zx^l)96molbjFdpfJ%lHfsHtq$LjZ3&=(W{Y<}VZwE*9_xBe& zhLXdwB->k<_xP|VsmDKev7ejeq;8yq4%2Xm>Apfn1xH_+dz^PU-3$e-= zo~mST^?vb~LZT9~7~2raytZSlQ6&!D-V~{pZNaZ(G)p(dOU}H!KDpX9BeRf;1GFpuVL7LW! z{Q@jQow5a*I654{=$^7)qkJ&HOoD3^d{EC&pB(HA5mI{hDl`2RxLVQn8DOsEEr!Ar zQEA#nW)O(ekvXAXiJ=yrZK2*`eBTFR_dI1eMMCYDl&h1f$i{LP|F7UQ^72D;4OPz+ zm$%JzQr%y-6O7-&6Nt}8vEVAEV-vzR2? zNOc$=H5UlbW#&8-#H6r48BvxrP^>sz!*R>hE_3Qtt^O_mah&|>nI^+~iLPHy#0&!$ zt?5nrb&fsdNkXa|dpwg)i7(~l{4zvC;0YsP^ig5#<4VH;r&TS8ZycF>TDQ6K>~yva zxYlBIV>UQZph9_F=#yQ*aBYJ9Y!ve`j0-nws`Oimb9b(B@{c8rd}!g3SdsM7%>Lq? z@o;sv07E@KzK@abvtvlsIt^htc&<%gl7wg7?H3;KK0980uTC@Z%zARHzZ?cJI2?Dk zJ(R*v`%3@FXm??s+Gb8a7nm{>^KM$h9!_NbW0mgB9NsOrG8GP)R_Q|GIdel^z^Duj zqsSQ#JI4aoK zn%}@?mhDPElrZ4w1}^TnCc);(j{S;akJui%DGtAv9p<+dSg!xb-;#SK-3_+@3?7j~+s($Qigp;d1{33VGK+PrHcf6#% zDwW}ipFu_GC`!8#uED!LTt{OmFc*?8v?HEWz4n9m*eIP*bHI8ME*FeBDLBaTmdx)* z8I-<%f(HKj)k2UfYO(xB;ON}%A%mB(K^U;2OU?*~s3r~*SO1@5#$$I+k zFbMS_=5=1=)QhuQ!`cHpHJw+CVrne>y3V2ucU#iEf2->G(EgNU`fLoUotMJ&j|PUn zC&7!%9ntiINmDe{6L3Gq$LcB2bkKXUx1<@id*tJN`3Pi`mz8-BsEJe ztTR_eDCt+lkm{h_%hVNg=CfEXOxTJGFS2ZBSxVW#Nd~vMC5pp&RI$8%Wxta zbLG82zP~t2soZ{ZRPy%d=)MG{QDNXW*ikRRt(ttHvNV?)+qlo4^F!$C3Pkap0f{85 z`dp0-i;mGlX9oD%_lC|>1q(X7(7J;+veQp{(_eHr{bkV&L7OASSVS@iexy7-vLvNP z&DI;#Uax(6T5JHO!&<%N$M{vGHIK%u<`>`SN@2o4SwLr6Uvz6vLW~9_*h(@ zjQ87n4{4=nfW_ByA3+rNlUyd0G`?O+FhAf-G~?Ht8_St4ObK|Z#e8qLn!;MofbP!5 z6+-&0OA;{HIghSFc}@aic<4nW+yaNPcwGPHgXw*Q5N59|Wer?I_7ev3)5SDnWkszx zvS#lS69|-WT~Aw)ZH$FdCx0k@h@PzG-66iu1if^`*b+mN+usVQL9%^dUJG|?I9MO| zNlGo{WK_#2b=p)F9yBs(uo%i={-q-%%WCLe(sgSh+SNBRJ@7Pf@=t0a_*#>!^piC{ z?ou6-$lRK`ycqt^ort=eJV501d=`H%hR? z`kssFBXD9S;ZHk~^N&ZD|zrpO;(a|iPhL`sz zmn27^wLv1nxQzP@uCRCVp^howwHl^Ah@_X)o##U7V^kaYKs8Cq9$&^GXfZwZ4bx%v zi6!g@^#W%TV6fx;MjKUCMWCR+ z7zt=={4zzO(jw@lPkk=dNKJ=}a zghFMO=)7n^%!7yM*jn}5#h#_isFH@Z&dA202%9tzvQ@U8TymO*fA5!QYR0HFRinJG z;6=TfWx`!Ns+I@!SfRd@DXrLKW^^j0spq$W@w>Ko9B(|lDpCM#vNFIuS`+35dS?V^ zRYBFkzZ8!yU+IZ|dm%f8R!@$$5Wd;n68|2)hWm&~wmve4?>v+uXM_uoT`}j>>expRE394}mgBGBCG}k*vJ*w|R88buQC<*}|sa$!|PD zf+AwFbCNk(-B*oV0+P~q`t~ug^F)SQ#0CEp+FESA*bIYhO^rPHNddyGYe{6*slOIbj>+8H+n}Wp1Q0g26exkKaYJUahL>%L$JZ zbZiD^OJHX0VGI21ymit%i&8#Ogq%jCubw}W7-8+)5v#9z+3E$iA%p%B-70Et3id@^ zr(_dSW}k$kcNR&xIm-KrjWRKup7uSJx?s zhnMJNh!uH7kmo|PT);mh5d{Iuz_!4rk*z_*GW*E0c;Sdd-sIO6uSTX;>*ncALO$mo z6$jB8H-JC3@qL+t1vCn85wn}YfExprVMY@*%rl9~*Z-nm74&1{Ok$kaX>(&&vOSnM zM2>ecoCc8??vL&eBPtn^$4`5l5!rv0QiIt<;#I|2y#6asx*cD*b)ei7)9*sV7ncsK zeb(pG?<-e391c*k)PLU0bLKQ7bM`Q`ea1e-P)hc1)3iSgyzpE@tBJW(>OQ&tVWt!v zK^y{b0#F_h$db7}y&gioxwgMC##mdx;k<`53xF0$feTd|}#O8|x!DNit+yekGfy7D_Z}|ZL6c{Rj>`QFG zXDLv7XS^;7fJ@DY%QaaVkb2$vggQu7MX?(IP~v|9inl}pNXbfpkN~-gxljOLgh9;+ z0N8j~!T|tyU;v2$0O&mkCIDcB3_}G3aRkGW0a7XdTe^4NT;I?~!;RO+r1h`>0Mo~p z)Gu6lV|rgA%CZEDfdq}UywViY7A6(W|zW%Vg#leF`kj`(ezF~XUE3CKj}Bq7$y@e z-dw6EfpF8JeOH&pP2Cc zY%bv_hNf){4Gvb;DSGDJO#&7wPVmqOUfmcQ(@)4W(AO7nB&HAo+4}gLeBo6GNsl z7x&hapl3bN$WZ@|A_kBu)M-wFQ!UZzeZ?UX)HYH84a<*yeiWVYjoCnvXSLOSSc1c>>onq@>h#>GhO2|8FYP2EghSV45qmCN$>Q?UUZ6`ngWjGoZxDEDn0$OV zdtWmO1`-bCwK%#8f{(!`!OSBfoQk8rahQHGB%jTq2(<^-yu?8%Eg^L?G(t!@tw8KATdCCM81%~sQ4@~0G&}%zGBOca?d|jq=Au=R@*d(Oz!$e# zdyi#H?>d7+6jJ0HACpcSjt}%2b|-`1nj)zAy)M0l&6k^$?FvkMlA*5S*?UqAK6AyIh!onrgI$iKN(!KZ9Zvx`` zL_Yc485*6WZBE#j2G>XybK!R+T}DXCXHs=nIguT?a_zZuhgJ@=WE+?`cP_MWmep-4 znwwxOy_92^f|c?6Bs$~10LtA;@E=(R7}aQ&azyRIr{>!uUFX{I!>bt-rj$U$8{_gU zfF{XXGmBq1|x*vzT9ZR5$ zQ%3O)DbMS*Aegsc%aM1bS$`1qg?Jr`flCiLFeci-^W0SWrnb%A4!a`{{WHoqSoXmM zE!9a`eTm(?u+?bLj)${Zh1b#-XjYJ@63DWmI!u#khuqIQP4q9#r}$~n zeXNDUmW9QXJ%u$JmEr~y&m3AXN)!{;oNMgZ45h;Gwby$OKcr*i^|mn2u8T{2N7-~I z1hS%C!rAcM^J8JG$5~blX4f?>dS@%MK8-%xehF+t2HdU5t>zKQU{%Y^_3n6Hd*bc- z!U}jr_48j}lhJ*mEcd9E zc^wG4}GisRh_% z?q!L%>h^W!WgmPU1#d?Tw9l7)ZEf7&mxyp%ebEfU!rbPwVi&*wRxw6^K(9m4qtIm?{d?x_2>LGK^5ryR2RqZVwC44TnR>5YDoZdicbnli z{#;?Kr>L!R9CPqC3iJR|;L3BcGZKGajO=V(N0m~-CE9%v{sRS&$x_0RRpPcO)u^G- zHXM$nyb44ui7>9V{sgRgWmLsXn{JV9W_`yBRO{v_!~QqNE>yhy9Tpm@@u zq-Tp+j}W_@-3;OsE^o!s1YY8nDF2{~`VHdS_k`!^qlzOF&svHHxEiBabFk;0P6=3d-QvQ@A4QluA#md%=N%)t1kiPVg^K?RJNuYOohWO5J-(5Ns#UR<0|smt;(*U-A@HK7Xi3jV zM-7uS^=JCgfr8|gg@#X*%NmwV3MGG2RfAEIn9%L5@}(z zd{r`}NP7-cO-?FCk*me~6P(uZF@P3!P}3)Nr93|E5M6Rd2l1NCZyGc?F!lt-E{5HY z!-P;8G;>#ah-MTeO|M7%^5Rpv4oS=O|HXz{cHIuZ$E+Ihe|1!9OuH3uO3-hbs;h2D)n^TMfQoT~ifH zBD}!YFKRG$4KVcE6sICL*g~C8EwnQq8kGbe+YEeCs{rz0(^kbWhiDz$g9&I7RXs@p zth*C#7*MT*!I8O$W`Q~!$;4WxAJae5+%jz!rwcrNnUJ~O3*+HcE#p`%POE-R3-JkB zX>w`bhQ5LdGqsBKApDvsn^4KFCI5kT8OLegWk`Jx`qJ~iU0N5_?ZbzbrK z#NXH+nx)z$1Q2ti8fh@G`t_6F+|7%yRU&SldP-#+fv$SPwM+^`UC8`$6cACwyLqM^ zYWv+hdObz*ET$*Slo zVy9&*iFQ$X_e6?Px?jQ*Vs!k00wV9xPqlnI|KjsKUt_BsM>WC-h_bUB9veCg0&y11pTGzj7L{N3H;}f1n zHjyFrm(JG%pV|?+J&WM{cpiNyCK9px-#uKd#3^4R^bY;&+CESxJJjCnNDj}kh{Hrp zbTBNmZg6Mmym}4AeNf4oS$*(%vW3;){7o5z&G6$7`d&TKT7=)^B3e`BOv6!Z93RsV z!4qn$Niq@~r!dwxZngM&2DI$B90e;T>12r}Ct2DAQ`q?)j%tWJ@pZ86 z9TG~F1iAVMH+7sYl)A43DHe~hSQMvJS+kCCt~{94gSd@GGjKurX7k30Z7iqk8T$fp z0f2&sB?#yLIq830ifo1MqYl`QR5!XZ_F0H(uctua@>B{$2k5$dMPiKUBm)Em!$aQE zeL@46=0Z)EjLo*%7Kaqtif z^V@C!fja^?vSb=Y0Qx_!?1ZJK)rPu0sx+AI&E@nRjC9hc8&;ZdHZGq>W zZO65?PfuJBEj9z!)Yow)4{wame-5PW#o}G8*aL#-W`kQr-PhJx1q)5^1Fz1&exh0< z+48oYu1=wQ0lc)*m*?YNvKF)eQ?bkS4Bn~Fb2*YqCSMhs7{F6qAFx}mS4{}8ui65# zr+H}reg!Y~11_|al`(5nHPVd?rx)Htvneg(lLUSIemb{T*K~E&!j^_(6I;mF#OEC9 zyzMhADXF|_LjY($=)w)0k~qC0G~&yVQ1b8jC9SR57 zINy%759QqM9jcUbN7_ifvL2{SlDwhIF^@QTu28L5AaE9c6>vms!vUD9zLDk4*=2V0JzQFb9-b-bDqUG4FYoor=900tBKjya)$L#2+Um_kCV#i!6&SzDuI|1+{O=Rb} z^Pn(CyeryAL7OXOfhFT#S)0`6`UK9xp9$_R(gBK+IM1JW)l^}^KG>m1M7_P#U;&Ij zQ00g`c}55^7)3Lg&s8EGh9c1@MQ^_l0TVzB=Fu*0XmehABL%6Qhd^x#>4K0De%;wv zfl_k}?c$|GAd)H3Swc6Upjor+1N=n$;Y$#rOya7?Q;(<%v8RrGI_{6^-U(?WXqKw> z90{Nu$Z579GLg_LYCf8@b^L|SV2~r4Ql1l+PsiZ+nc!3LQkok;iQR8=H8QCn{{!5Z z@3F1!&eU`C62H$$$V%*>=uC3z9Hf$Vy9q@ zC%bD()su0RMp zy{Q9nglC6DCJhc@wN*-kk3G)9!FA)ycvTes2C=hLFfdRMikS}>5K3abV(wxlkp^iF zMQuL{D}zY< z5r>=;jXbEQ5bcedYYW%YJSEV2CD<$Fy%gjFRrb@ZmLwv^T4 z>VC!?8u2mbmy_6@nth$BFUx3yp$06481wWZxwN8jbkQ&wkCSc6Z_2GU4n0RH0d%C0 z+gq}G8q?ih=ZIc}Mu`k0%DYc$>FR-oyIh~j(aSemEKbP33y8A1_U8V{(B{XKW32{( zvVG{)%3v7I(Sdn3d@D#UOFVJxYM+sZxUJ08FK2P7$1{T#jTlCM)GbjVlIKs{U}?J}s}^ zIG@$ZTeG*#4nHS_nK~5BP#mNuQEIQFtAs>CBMvp6;rcm1 zRYTP#N_Uw>ZI2K2MIM*60Z#=61}Paal3(H&rg+P=xF>p|V3ZGpLB={4U7u!XKrs|Q~opU+}Hiug}xUua86>vCGOW^SP14?GqxDTa@z$oIwx&1XK(+1fOv$wmO)KWCPCp+}L`-r=g6(LCy% zQ$#X@7ATyPo4aF0_2)Ks`@b0>e|Ksd$?dogZW<vccVGP@|8!#|@*0vT*_lDHKeoPtHamE3?Sii*Rp6}P z$BnjxAgZI_(FR^_)&1u^%uxlr4dI;mjVul463U!K*w?$KZ3)B5(*HqY*{Y4YdzCN3 zKV2U&ht2)}SiZ=0wn-l=}MnmYW|yxOZcx; zS>*p%XiQ}u^x(6j6l6;bFZMqtNV&Q;`Fnzj8gHR>rJA{r0I{l9Z(Lm)xqfW1XR%$p z)c+HBN|WP@8#!(ViW6JAMS&o=HlhL*qZr?i*<#byFza%h;7==8^{N6s^>aM@&kcRM zy2h8{pRV12eZ6_wmK6Ig?QNYwuSc@p5aPuW0cK$j=@z?55h3}XyVwP;P<{EyJ;8-s z3dCvyfuf%cdX=rKo~RfVR_eEPaPa(zCg#tVEMf2ZC6rlb(T>}W*k^f1J(aamWQjTC z&hJ?_@pAY4sm%ILFUIJMPHgoT1#HG~5M8mgPMI+ruU}pkY&G6f<^NAMdQ_xZmu`$~ zk1{rFN}!VoEBS9T$C?+l_|+b&n8?Q@sf$7{fhR1?fKYK}|RkXTf? z|D8FGeO2DJ#6Ojuw%fB_1 z`i5}io0@}3rLzX11U3GOjP=jRKlp?3!vS~7%gzA`4xB!-XY`v@0_mVv79A+J^SY>h z{4bXba#{PXH$Z_&aO)u^mo$jair=+aed$KvyOi5AHd(z-^QRB__a49mdRg$mekZqB z1nsk1%Vziw>w=nd=P>$4ddoGpgXM4ee^|FamXL-x6e~zRmDR>YSL#IjkAtb`0{;yX zgoCN6Hw;x%;`82N+FM{f-aj7g4{n|U1ue5o40>^SR1@ASDy~I3+B1Gt~ z9dbpy<**;q7;p&79$Q?wba?Y$%q?#X%W5I6DxS?WBnW1B_D0o^=iI5v&57}=2k9Rc z_g`!*EQ_<9yyD$I4x{+LZT&YZ{uei#XWsv2egE1(Oo{A&GiZO|8WL;$r>XyE;{QYX zF2woY2gon%R3CbhTquEzpeyBXC5#F{v7*92E7^TLqHGxPTc`7X1f|J_AVEZFJqRWj z5!!}+jZF72>&9usA?*X9y;qh6xE zpclc9LYR(`sAc5;uG7^V5MQAUh&{woo5RS5G9)sz+WqvA@LI>U1;@FM;vp)8vAPEh+yK8h{MV2j^Yag184B% z`BB7$!Kv5gUiXCo?#{!(p^1qJOqRXXtD_4qxy3rFmP>j?3CUt=Jlr~yUgYYvZViih zXGih)d>Or2#()pso;dXr*CI|#(Kueh8134Dc^y{HLbHkqqHHzc=<;qUH@k7bG*r~x z@v*^jr}$XhcdeuX^^V}!!?cE9$hHw*R#LB?<C=OKaof6gkeG9Km~fMp4^#wozUD!Ex8E4g z9&&5jFXqQv20|`=9U$OdcAOuNZnyls=686trXnL|nnXBFNpGA+e-sx6xqT>C0&Q*) zi}ksvN~|~MOb_JCjJTgawVgtD!WF6>h7ORAl9 z8hlas#aueapKA$9}WoEfthJi;oy3Idw<7ue`w6>5yhl3BVe z3-Z}t*+u72`p_5K$I>z9+@DM4RH(yT&6RpKZxK{_$c2-{B|R+AvGh$A=jhJqt*%xj7{P z@BJeJ437ZLjNn2$&B*89^qJ%9PIa*Oo(n?yUs$6ICoLT=RUvlaW5jQsKWBVlKLGOz zMDg8H3_r4!UH!2ffupsGN*^H9_oS&&Lg?B5T(iE@qoJ6})wb!zrc3NFP?cFeQ;0z`eA6wDqatE$#B*S2PSuo+I@ z)WKy2&v<_pENY#ZejLv|j07ZJkxcKG8m1nZIwnxLkPrt`&n9*gk)2c|ITSoM81}4% z?@Ij0H*peIsV(W8#~C%#B*WP&S)C^hWC_KDC^nh~e=%woV^!j0E~#K)mKfy$Jv#FF z)trUgw}ih`|I#pJU|BhtJro`yerLW!AA3B|Nru~D{E}lyT1^%c4Qso6HrGXii+a{M z4603)bxJAyPTp$TFd_Y#5Omr(*m&uA1o`+{2C;YKgrbk$R1H?0TE)I)3lt z8oIq4-@;p&C@x1+Yu^FJ}w;tz{`rXU{y;ZPmx zBO4wFB)QQX3^N4%h-0l%D>Td!)ca%C-l*~Djpfzc+u3JNGf>STdtZzE+{Eb+yJ#lK zr6}?&`d2rdNbRa!o$9ta_t3b&UPja^qcw{juccOf_U^XBk3|-xJwdlJjtB1B&`&SGBjXE-##$7q3JsQj4@8U<`rv;pq0)r_q1!NODfq(2ZQ z%e;8+!OSlGZPh7bCQl-+&Jq`y{%Xa-CPjl&G~n!V!ac)LBj=+cnpSi-{oqbR6Q@8l zJh?m#^4*})ct(`ahDdpk1F7Y z61u+Uqh@vBMh2z!FQK36uv3kubwJ@#`v+Yr-In2HyPEnsMN@2N6O^b)U38t!70_mN zpNFgHhxuApx0-kB3K4FhKw6x?V$jZ+e~ajQ zL?`9}^^KR{cfydpKNg`Q+ZP%A#0<=zb3 zz4^G%>HtYRrO3F7 zLffQz{#gni?6%eNkHeG=4eM<_k9$R(XcQ{Xz^uiAX-gJZ_VaBQ-`4h?r)ORLUUJ#4 z>0X7td)0x^_#hc<@GE0=q$TUBt(qRPvgJGu`%Ds>YsSy8HnNZP<{sa-G0-K1*4!VQ zImz&rs9dBRc$x;aXNGO3SzaWle#a)q?c5XVu~2+I?Y`DY_RKUUDS3FW z&V&%nS|adOV-_puYty=4u21sT#BSNn1N<4E+AGrw%Y$o-u&VDSm(bk5bAwe*yK2mi zWO0aMue18shErTfMLa-s+|xeFS)5tZNMDoo=My7dz5ULOUAmb`BdvM|aF|6*UvpsL z5V;tgGS_k*QkhfSpV2;o;*pPBxHzPLJpQaQEJ06O_k-#&rsg8XRjpU3R`eFpB=D1! z)hR>TQjI5-B{Ld8QJW9V+`$NJUf>LPy6w8|I=#vksN&M-A$vnU` zLd5UU#8x1x!HSb?F7pnp?pkwW@5ibf3RC;6Zsg*k)|b_cbpA|D4oPZp%VCPmKGzvzWI*m;H}zznBP zfGNk~Sy#vL=oZS{@Zk~KDmMFuCr@LWN8qo5=iTM!a&H(RN z!@Gz|WAWDag>)Fcz-V;dU9o9_;flp!-^u1I2eDVR)g4DmJEpwrWnT65d!>1A7a;~# z6+X!AbH$efQ=&CdCxDJbV`oYKPe-pV0?xO3L|;U_n{=_cdm-_!d@rD&&ZU2>=#e^u zG^gAPcCXcZLjT4qjbx`DqY~<9PXb(Hozh(n6M9|;ec{<^t5*3AH`PN#&iUmDB<}@C zap0=Czp7vC>=~mj?0x7vhL(0B$kx9B-CA(MU>)AMBU|}wXWn@wrGEa$y6>XkS>8fb z5W(yzn`J%KcfGpTK3xt~KNAA&o@rIecz5CQtL{3?a4#nCw{Z{0DDAG{5^}+vavoc$ zm-#1JIFr$UR4Vg#^jHp}$IlE3`&GweOGet)@>;rX<_XF=7*0Ij;E?}V9KrfC>6Upj zjVmfzN|L;<&C`i|=vNgT>-Iu2`ZV-Z1iQk=T2{W?EY;{yQfiZR?Lbqz(SA5i2HuxE zsy|8mcgx!w`WA58l?KfJF@unNq5-VG>J4G-P4zFD_EyHL5vg@-txMbd^b;|LUW~a; z&|d||$P(1*P@D|)FN-Rki5}t^Qy=^t)rfRRE;ZWTj%9gWCgC0}&VgU~%I+?F_WGY2O}Q7qw+VP{82#{{^9 z9CaHO3>92+@Xs(8)t2k)u8A!A#v?BOqAp$^c?xrP!-S1sullB3HV(m5ZB7)d%kn|O zi;u%-&9v_s-_3)s@~%GO?!1PFXiT4#iP^Pd<#iPK3pFbypli&2T3_#^Ea=cbU-_BU zRG;v!h^&Fw(zUTYX`v6$f#Vq8H818QmmOi!Uwt|CB6i%GTpA?J;&`b=RcjhPVm2;8 zA(y@4xV_e!@z`TCRM34TK3O$tZ`JFmo}nFATe$q8(|jFEDmsjRpfV>Z`oYuRgUM=5 zlkWx}uDt(IXqH`daw2^_&2U|oXL&i+IU~*?F7#4F(H`g zh++v?9n%VfQs6@eE&0vLGtGIbbPdE|0AZTIpk}(hC5=$$b!cUM3kwRm^Pc3dv-6IE zCa;yl>8)-9#P-*(`oo;->o-qv>+0)$I#+g3d|tI`&&^HF7#QHzRa=YTR=D8)Q@8qO zDq?8L=7j{Wg4U@|Kvy54n3>j|En)XK9oLJbZ@{QNmo#E+Uo@Wgff4sKRen# z?fVm$|8V7VODefKV?#5Hf5=)QR`7sQz_BAZ;t~C@uVUBss>1CL)$P_E4I?Bkg;Z(E z^JLezWe1zoj3z!wXEI){b#1IDYuU8OaHRf==7p^q$YlJLljQ7tKJ|Rva%b41)7+lpbupoPG8Z`)F8G`1@5E);On7TOR3pc9MJy0lmruL*6U7; z+x^oY_L=c>PH!|)!NHQ{4gU`hY z3aO4dE1T@!NdB%rho3;o!iGMYdw7laaIog+=>`GocfN7o3D_hKWcR}drt>!u59Ry% z+&7E)CWiQ^ilE0j=6vK3(T>}abKkRa4E>GQl-aOXvET}OSWC*>C-qiGhJ;^V$R)wu zcl~1%R|IhN?r#G5N_NYU5jE`p(c6{BL)}IFDMf{3Cs{()Y$+-t5!u%v%MeA#t}%8M zS+dpGjU|c3D6%s`r0f)?>_c{9>|>wzH`Rld=Y9ToKcDySxxahQJ?GqW&-tEn?(oy* z@jR{pW*>)CqCTKHi>Bb#R-D(<(b3amidqnoCtnfa()gV%RtLdy+5O$ zgTBZGRUm#4{WiA$)RQ!Y1@&hsi>gxQktD!ZkcLZ8t$t=w0Q=VAu~2t6xp;D@GA7Rj zQ<1G`#;Sea)Jk;pWd`iP3%)0t*CyRBu6-o|7nNM3(1lc2TsExH(9zF3xrUDpwFn&=gg@(5)35`Ub z!e1!gI~T3P?OgSndFpvLD_(kEfNR2aijvRkXQSH|n`|60*77QHP+6Qz5tU8fY3Xa6 zK3_dUrhKYObg4SWG|o2HeUVoBAHPHwoqqig$3W56TU?IGy9Rj z`T0QOZOH=jo|3lWl47&5FO4$w6Z^ST*LtNRHjKMU{T@ulzVm+dYzW9 z8rlFc%OYF2n7+TFZ*Y7D_m=6f3KY|IHDXspWx?m!`8Q3+Odss735*&_ZaVfZnK~!qoOXX(ZtB)#j_)dUzxXJ!_6OZx>+!llHaUz}IAbp7UKM#}+8Ekq{#tm^p07T!9L zWN$>}5oRRz#Vp!G^0IxRO8e>LbUZel`RFgfo0#o;6F=>-eo&E(fO5gzhylMytMhvk zH2()eG$?(mxNw24d#D=>k{A+ms>hArxMNQ$=y=i9TH}bKRKpZ;bQ0z_emwC5`rAppbu~S5e z4{NMRH5sR!W{aw(G!xk5Q#Muow8vW&yK2uuoSMgY<$TR!Tsc6>hdUplM%*Nhsyv`~ z%JGs?c6)w4Oi~DI_j2btJeP^I_m7_zX?(--R6_lfO>FC@4Ovt1BSys?3ZD~ytObU_ z#JgAdEE#wG=nr&7_Q!rxj=DRF@io&-I#CGWT-z z^%suoo`l3WiVIN&3?;TDco=OHNkO;Jn!@#DANfF*m1bJ6RBnXMUAdi`Ivd;>LH3(| zS2c#&HyX(@z2ZEa(eW)Vf723fm$2WU%g6S)fQLhDUkN)CgBI$weXMJXX5kTwMaqWU zS1{ftLD@@Z$=jza`9gaOd#A&>%eI`nLTUPX?GoeF?GlGUhVk665~ThWQiiS!OE2+3 zqg`S2D9ptFE{yDEj+34~LGOcuC z{Hj_pYV3nRsBvMhux0*-D>7lyVxdQFo@7^cU|-0ziR{>F=9o5WX6@KXhTBpX4PW^o zg5YuC!|=x5h7SeF22{#&f_i|ebZx)h2)x^1GERsN7mYtj_6m(?7%R|ZH-n@5d(|8& zy0+$$&VIJ4FJE$X9UuhW{UY2GnqPXgT-#RANOfI6m!@#4edtPN(2Puf)E5WJrfR`Y ze_L8>74M(cNC9t@ZphZ^ztur8=*#(v-n`u?6D-+%wuJljXP%Pq*O$`v6Gkmv%BCPX z9$fMHT^0Q@_S}d`zJ+1#A)vz~gS~;TfGgu;r9AZhv+%0EbT>QVnO7?g8y6ipiag_= zaYh$$1k?Yxy!@IRdyh`clu8SP?Rrr+w<@dZ>EtcP=WC-|+#aXi?~8svJF}v%F3%BO z!q^<0A*Jl1^yt*b%97G<`5k>}x@21yNIY$ubucw93VeQT!4+r+`F3}{3P%0TK#PIG z{$>*D=s_zgU2mfVeSy&a-Wf2wub+YKx&PtDc1C%eEZ1}z`tqj_X0zH-5+96=qa1#W?(K-;g zg8KXZ65@wk=GbgRqmD{m(1gH@*F~AH_jb%?oL6TA<)0@fM=!4Ogo|hDv(s*lCG7~$ zn2a_-T{}|oy30=_wp}87@=pFR-AHv^<-`ovrEP-*F(i62$T?g1Rc>`UT?|nOId%~b z&}&2L%VFofI$ZqHylW#x@JLn`i<~OHMTdJmXJeilUR1*Q#V{YlWKd0BSSci!m7xN_jEA8faICs#2f1S(cNd?1Kl9173m{Z^ut3C-` zNJ@!uu5II2hMAS5Z>sFYm4bMAY5jLcz3T#thDLd?toYej9}|*A8D$9+?U2gu!8THu zDgE6to z2XJ=q{_q#MuwtB%YQtc2^Ou)4z1TDPVTuW%aN?)M4?6r4PT%rBSAIup&Sw^ z2MDVk$SGf6%PIRg4=-JeI(w;VUrwUX)6)*U4ykUYZ~Bx|S8urE*Sj@jZj~>7y}Hfs z#nei6k?q>vOSM>AiuC6(tCOmkdyOeD1FHXmmeh~c1bykHGRwEFP{E#h0+{!*WXQ3N zRd27}&-DVxL|kgcwK^xPLsjF|+2s}gx?Yj&Toiw3-6w|@kM5~fAl^C)R4;nX^O-r~W4J8LKs+96k+{AG0tISXw>bA=ugdPkJ{T0NvNJ}U#n_jzisr> zSLsT5Hzy$LFWIMp8-FmR5s#m-5(PZo1c`0bqaIt#*5!*v?z5Au3Ec)_oEvthw;f~yEAqUPQ1{SK^u618z9M}>S6L}HNUB6sW>$NAoeQ~Hk` zi5_GT1FIIfY#=J*!1Y3=G4>X}>cly48;(yY`ZZ_X9MbQ2{kDnQp9>2Pq3=)dZa z>ly5=??EA1pP@IAk2eiHylBYHT_hLsf-{^{-MrStOS#Gx87`OumAsvLkzI^8KQv;_ee9$dzbX|5y zGgDhI(RTn6wp&sKzB&U$yGmOP=WE=2BcEs0P&ujy>Ks>t$hu>~XD(2@1KL!b=K@<5 zkIS;CO1=G)0C{knV-Otik*`dJSEx%sGvC>?-KsL5;mwrit#4cull4@xH}fVtb$%-B zp7JVy{W<8!4#rm#L89FP(l4sVs;R!cyZ#i78DY>{OA9bZA$Urq%X25~_0Y^Gn&*P| zW+%YfQb3M7Q%B_jctyh?3xM1QBU$tx`y2EFdM&zSo5+7mkIkK1@l?;g?+4}gbvHk$ zfjL529KJje;65_u*;5=QMw*M>ruY%Y2!-q6wHNP0DhPjm{7M@F*YjO(CldncJRFaC zLbGUjPP`qoBYK1+@g;oDO3IzYAY}s;wa%_g(@W= z?MhEEl5i}{x?M_R)l(3oA+)aWN%2lTKL?@}s_sgPGWGUMc2mVSi&{N}r zn*+(asRp22c$HIh&z%01V3=6YNshe-saOLl3y-hlngkyP>yLWC`+VR&2Y0r}oQa;K zKkK{X-SL9vZz^S0y_pS{JY{(c@cTm4RliN4zT9>JKCl(eS@%WLwH_Qy_*ke-wCm zUk`3Id7^^A*5Y(|k2MyF)D|4mzYaXg#GH(rJ=_Q$O*cPhD2p+HD#IEYmr=JQznf&h zU8_Dx=pAdWK${R(8>np{=VwP4jgOC8bmpC9lKA{U6`m!9b*aJnxT8 ze?7Hb;O-d$c5=3{Ii)NtBqWrqCf@L#RyD3wWcg^l(pq;-p6j?qbmM>%$)WV*@hF9s zXQfHpLH&AhA`Pr@3mo4snSTRGUcEK~Ugzx#ImpIgx%P}|^I(Yiqp(HchPvgBm4Y=x z^xYfl3Ff8y0zS3AQ_fEeKlV7lba?q9a+Z@mTX=o+e*;%aYNqyKF4-u=DbXoF+Y;MI zqw0l)n|QzGgO#_LM>AtXt+ekCksi~R%FQ-eKJltu8yczWtlj88@}HAJl;H&x>ElOl zp`?J9V}>m&QdlnyhU)W9OtT$MS3CYR@lNz8Tf9_lPdV%{k#8@47RRues9ve7&&q)iOaO>?20u}d?Y)wLWV?mKLySKvTlO0m5{t3ZKRy0_ zGtQ5PUW2lkaHc*s2zyG-PuOz0(pG0I5QT6lx<5fWNap`{b)t$8y~S7vBz0~iV$V!I z24U1_O?2c0qZ4~pXy`MfdrUQ@YQMCf0K;a0H!qGTG8v`qDJcGqR|&oZp8k4il7m)S z4u6+bi%Oke65=20ERI7?Y}&$^W~3b@yR@KjvBC@eEnf-}?7SL;=5YT;PI=a>@)pM# z-zAFiOyD71w}AWOswSDTAY$O{s-zK{TGz()$1O0s50Z?@zZ(I zeIK=;*$5$wMyzw=b-*k4qlMJjAmQk z9v{nTSQ2N%j51fQsWbeSQrSvkE=h>yG%;3uK%xsX!P~dS<<0}oib<)y>XZMErGW+> zUi*;SjC5n7@nz)0x)9$3t$C`GmuxjqO^dB78b-7~97`Z3^!kp}?RLcw4V|$$lmwF@ zFgZ-m&^c!#F?z!r)r3H!Q;yP?X3c5~%vOlB!oT~M|AJ5g`pSZI*w#PnD&K>$?)6~4 z!SrH^EmJ3R-T~DlX`dAL^}h@N`uIL*)ETmv703Z|Xrd zOw%?i2Th=H2Yf#o0?*R(hzsVtP#+Wy37VAZF1W(#NvN~>ps?4Y0KfzQdw1Nwz3En` zwS8pJsP;RN!h%9KQ_{PlsA>~UDvrOHg5R|YfyzO}j~ZkNn`x@nr3yOZ z2R+04*;%R4?R880(k1-%d95?djeH(f3uZm$D)>^yEEk>C-a~g+%m+D)S;A1Ecw|TP zA6)=0igIpu6S+rk_9NsFwdhAxXyMcE6Xw0RK!PotS`-U z#d=|FX%g;A=EWoi5k>c7|*RBI(=G z+V|(c;-w)6xZNActKSFm80WB=$Dd!TEZ<1mk94@V3)_|{0o9nkzDf;8mEKYv`|8_OS(iS=LZx#u@b`s)93Zy`&#&t+x7KZ(&Tl5!UR9GhcBX~|LwRg3 zVj-m4FWEUAdb@&hFRpMVx?d{ki-V(V-!&RGgQsrT9^Wl1M1&^`yE^5cY^Y4raMTgu z{m$8+*bfEZB-$lv_eD~*n9r_oWf!H^lW3|R{(Yxfkd)kwL19Z1yll7S(Vy&Tg(-V; zJFBzjJQYbxNF(*?^Z#w140;AY@4?pRv&fvx1a?zxT~+6MCCV;e=$@kY?Iw#y*@}R= z{nCdMr0Hlj9K7CUjI7aGkY~c@8oW0X^gA1nv#sNrQjyATIHja%&su1}kaoA;ljLN^ z-d;^$+TC<9Ff9p6^_yK`yd){+lnZ&{+Ry;oDIda8?4^D{v%~gk_AI!+tyejp8iy$i zPI>2aTT2-`&q|8`$_t*?Y4zXO1u+32iw3i$(2%oulj(sr#Er_I6F&tKR0`eb?ZO#ibjY!{crEg2=Ytxs!p~fWskPj<&A9 zDugm{tiINid-bhoMvbHxwie)@qN{%q7w|to8gDr`@WOH#hz4-}E__{LC2hmeYsBzvR(IqX9A72Kt@)wRi3pO>2juY|n#nvXyy zwjAvOe=qC%fi@{Eod^th%KA4?ga|RRV`sT2XfI3t)x`kPN`Bts3;=}vBmD&QU;k`f z8vT(E<1vnyxIjTGC-wIKGhem~4gB8|cYPB;Q?tcQ;PwrO`vq~`b>RuCA@Du-*|#`qF~ z=~#xvY){caI9=Y<$IhP!?A!OxDCjpFp74Y-_&#^+0R(xSC5SKIoja)A&Ws?WDae`%^X7|6OJR@-HTUg>Gy$nYenTlv09i_ftwI}n2`^TwOLUbN>ADQ+g zi2UThMTL=gfCK-4);la@wHIy$MShzG{uVAz3`oab`E-)AF6=KU+v(K;EFqFkj*cjn z#}>Lov3%bs?uYdug6TVs(|oQMJE^ISGa{f48KZSKxL#~{7v(c z(jmSQC4f*GDV=u!&BCpNzQZ-=+rRcu%r*LLY+*CH=!neklIE-nhmjBw;GIZvNUe&H zO(%E4d&bvuXPPHJz}y{Qw)6o%#i#k)Mcq0gJxRw{rrYQvAdDr zT$5ufNjZ@ZdCm$EtiKC!&M9JP!%vmEwg$gszTAhb)Fv$L~YF z6MdeuF5)jd4RQtcuK$SR__lL$-ZYT`EJ`C?e-YE1R%K~74i|SOI4~8$o8lg3bohNH z^#$L*h+aCHeJ;L^X-%_=fRo{utRbquM0oxD-vp4h*^MJ`Ts=nZhu^>|=TU#(w~ays z`(I>AHJt5B;2~w=909l;M8`7pom|C{19QJ?m9E&)+JUsq5no6CL-$1*s)gH5KbxXy zOy)}K66NH*2P1KcQ;>>&uNOv=2je5$AeDcvJ+(;#YN&wMN65Y?tGU;dAGCX$AZXv|Vqz)B;C>}6UYI)cs7kz3 zs6x`W@}cN&D#yHlF`1ODh1yNdF$WC?4~y+bO-JQ!Y$4pAdOY9!BGi2l6z~>T8p|M( z^v$B39po&Rl`~C3B6nUZAAs3Ojxkn0rlQ3FjK9Jqm@|#0yfH((_@UW_rnQezA@PTSU zvauw?ENh3qo$aQPln0gL!xVvjH<|30AR<-~^hK76r1vGD0i6W)c z=RVHL&hdhV@^MrYQyPqfGT`@zArznkO*n2eR+U-pAoj6p?qDn5IeNhq*Hx(k@cL{^4^Kg>EvxPOhVuTy}nL*}x zskcQg-+e*H)?0BO<_|o_C9PRPi1G&M)|E&CypB)afBF20GP!4=-Q>E>PyE(QNOhS( zhYqme-%0mRj%8Oh)cN_HN(F&r*1jHDuJfyUqIpB58!uSKoDL9sj~TK`i=Wbz#bUWN z&E(MJ_sl8>DGuQkn@O#r(+ISRuzhp&RNhZUNmF3rT;GOSc-AE=VkF+Gj;|~koNZ%0YweV%5IyxCLyZ6R{M#!NmljTn7|WOsd0{EH#WVu zu-F2Pl}kBk`z)|{H}BLk=g~NErHC7E_uc!{4I8PCl!dG*t?O42zEDjVa@-`8RCQ_l zJ`9uF2TCe~^OW!tHrFpHGQ8z5^Co1MnUR7-^|!ZY$eL+MV!WM4g2eqy%4$l2NkEE2 pV=;mTzU%Xc4!}s`5gS$eQrGgBtkj$uf!BvY%2zcMa<1HX@IRSN3g-X- literal 0 HcmV?d00001 From 145f506a0085c6a10722d53f0b49a406f357151f Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Thu, 9 Nov 2023 21:19:58 +0800 Subject: [PATCH 2/6] Update 20231102-intel-gpu.md --- rfcs/20231102-intel-gpu.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md index 91d100f..b3627a9 100644 --- a/rfcs/20231102-intel-gpu.md +++ b/rfcs/20231102-intel-gpu.md @@ -3,7 +3,7 @@ | ------ | -------- | | RFC# | | | Author(s) | Teng, Lu (teng.lu@intel.com), Yang, Sheng (yang.sheng@intel.com), Zhoulong, Jiang (zhoulong.jiang@intel.com), Jianhui, Li (jian.hui.li@intel.com) | -| Updated | 2023-10-31 | +| Updated | 2023-11-02 | ## Objective `XLA GPU` is a mechanism to extend new GPU to OpenXLA as in-tree build device. This RFC is to introduce the related changes to integrate Intel GPU to `XLA GPU`. @@ -20,7 +20,7 @@ This RFC allows user to run their applications on Intel GPU directly when use Op ### Overview Below marked components in OpenXLA will be modified to support Intel GPU: -![image](https://github.com/intel-innersource/frameworks.ai.intel-extension-for-openxla.intel-extension-for-openxla/assets/38638514/b628d3ee-640f-4e2a-9f3b-22ac891d7000) +![image](https://github.com/Zantares/community/blob/tenglu/intel_gpu_rfc/rfcs/20231102-intel-gpu/structre.png?raw=true) Here we would like to distinguish these components as 2 different priorities: * **P1**: Related to basic functionality and is covered by this RFC From 2ee398f5c7665af5334d937527d041567532a11f Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Thu, 9 Nov 2023 21:31:41 +0800 Subject: [PATCH 3/6] Update 20231102-intel-gpu.md --- rfcs/20231102-intel-gpu.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md index b3627a9..eeb68ec 100644 --- a/rfcs/20231102-intel-gpu.md +++ b/rfcs/20231102-intel-gpu.md @@ -1,7 +1,7 @@ # Integrate Intel GPU to OpenXLA | Status | Proposed | | ------ | -------- | -| RFC# | | +| RFC# | [99](https://github.com/openxla/community/pull/99) | | Author(s) | Teng, Lu (teng.lu@intel.com), Yang, Sheng (yang.sheng@intel.com), Zhoulong, Jiang (zhoulong.jiang@intel.com), Jianhui, Li (jian.hui.li@intel.com) | | Updated | 2023-11-02 | From 145ec6dcbd3dc5a1409f5d2518b12c7d3510c3f5 Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Wed, 15 Nov 2023 20:35:36 +0800 Subject: [PATCH 4/6] Add note for code integration. --- rfcs/20231102-intel-gpu.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md index eeb68ec..274d81d 100644 --- a/rfcs/20231102-intel-gpu.md +++ b/rfcs/20231102-intel-gpu.md @@ -34,6 +34,9 @@ Here we would like to distinguish these components as 2 different priorities: In future, we will follow community to enable more **advanced code generator** rather than `LLVM IR` for Intel GPU. ### Code integration & binary release +** **Note: Below integration method based on macro is an intermediate states, +all code will be merged with other in-tree devices once the software stack is stable.** ** + We would like to introduce a new macro `INTEL_GPU` (Tentative) for code integration: ```c++ #ifndef INTEL_GPU From 08f47989b45e56cca5a0c33bda921209202af947 Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Thu, 30 Nov 2023 11:02:03 +0800 Subject: [PATCH 5/6] Update 20231102-intel-gpu.md --- rfcs/20231102-intel-gpu.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md index 274d81d..f393210 100644 --- a/rfcs/20231102-intel-gpu.md +++ b/rfcs/20231102-intel-gpu.md @@ -3,10 +3,11 @@ | ------ | -------- | | RFC# | [99](https://github.com/openxla/community/pull/99) | | Author(s) | Teng, Lu (teng.lu@intel.com), Yang, Sheng (yang.sheng@intel.com), Zhoulong, Jiang (zhoulong.jiang@intel.com), Jianhui, Li (jian.hui.li@intel.com) | -| Updated | 2023-11-02 | +| Updated | 2023-11-30 | ## Objective `XLA GPU` is a mechanism to extend new GPU to OpenXLA as in-tree build device. This RFC is to introduce the related changes to integrate Intel GPU to `XLA GPU`. +In more generic speaking, it will support `SPIRV target` based on `SYCL runtime`. ## Motivation Intel has experimental released [Intel® Extension for OpenXLA*](https://github.com/intel/intel-extension-for-openxla) based on `PJRT C API` to support runing applications on Intel GPU when use OpenXLA, @@ -15,6 +16,7 @@ Intel would like to upstream the related changes inside **Intel® Extension for ## User benifit This RFC allows user to run their applications on Intel GPU directly when use OpenXLA, w/o installing any extra extensions or modifying any code. +Besides, it extends OpenXLA to support new devices based on `SPIRV target` and `SYCL runtime`. ## Deisgn proposal ### Overview From 49541f1e8b0bdba6ca003797a5ae01f4a2f8cbb7 Mon Sep 17 00:00:00 2001 From: Lu Teng Date: Thu, 30 Nov 2023 11:13:58 +0800 Subject: [PATCH 6/6] Update 20231102-intel-gpu.md --- rfcs/20231102-intel-gpu.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/20231102-intel-gpu.md b/rfcs/20231102-intel-gpu.md index f393210..4ab9766 100644 --- a/rfcs/20231102-intel-gpu.md +++ b/rfcs/20231102-intel-gpu.md @@ -7,16 +7,16 @@ ## Objective `XLA GPU` is a mechanism to extend new GPU to OpenXLA as in-tree build device. This RFC is to introduce the related changes to integrate Intel GPU to `XLA GPU`. -In more generic speaking, it will support `SPIRV target` based on `SYCL runtime`. +In more generic speaking, it will support `SPIRV target` based on `SYCL` runtime. ## Motivation Intel has experimental released [Intel® Extension for OpenXLA*](https://github.com/intel/intel-extension-for-openxla) based on `PJRT C API` to support runing applications on Intel GPU when use OpenXLA, but in-tree build is a better way to maximize the capabilities of OpenXLA and improve user experience, Intel would like to upstream the related changes inside **Intel® Extension for OpenXLA*** to OpenXLA to make Intel GPU as in-tree build device. +Besides, it will extend OpenXLA to support new `SPIRV target` devices based on `SYCL` runtime. ## User benifit This RFC allows user to run their applications on Intel GPU directly when use OpenXLA, w/o installing any extra extensions or modifying any code. -Besides, it extends OpenXLA to support new devices based on `SPIRV target` and `SYCL runtime`. ## Deisgn proposal ### Overview