<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 6 Mar 2013, at 06:00, Mark Hahn wrote:</div><br><blockquote type="cite"><div><blockquote type="cite">The issue here is that because we offer 8GB of memory on the cards, some<br></blockquote><blockquote type="cite">BIOSes are unable to map all of it through the PCI either due to bugs or<br></blockquote><blockquote type="cite">failure to support so much memory. This is not the only people suffering<br></blockquote><br>interesting. but it seems like there are quite a few cards out there<br>with 4-6GB (admittedly, mostly higher-end workstation/gp-gpu cards.)<br>is this issue a bigger deal for Phi than the Nvidia family? <br>is it more critical for using Phi in offload mode?<br></div></blockquote><div><br></div>I think this was answered by Brice in another message. We map all of the memory</div><div>through the PCI, whereas many other people only map a smaller buffer, and therefore </div><div>have to do additional copies.</div><div><br><blockquote type="cite"><div>it would be interesting to know how Intel thinks about the issue of <br>card-host/hostcard memory accesses. </div></blockquote><div>I am not authorised to speak for Intel, so bear in mind that this is my </div><div>opinion, rather than "how Intel thinks".</div><br><blockquote type="cite"><div>my understanding of Phi is that <br>there's a DMA engine that can perform copies across PCIe. and your <br>comments imply that Phi ram can be mapped directly into the host <br>virtual space (including user-level?). </div></blockquote><div><br></div><div>I believe that's true (as it is for any PCI device). If you want to be completely</div><div>sure, the sources for the host device drivers are all in the source package that</div><div>you can download. (Go to <a href="http://software.intel.com/mic-developer">http://software.intel.com/mic-developer</a> then "Tools and Downloads")</div><div>So you can find out precisely what is going on.</div><div><br></div><blockquote type="cite"><div>can code on the Phi also map host memory into its space?<br></div></blockquote><div>I think the aperture size issue kills this as a way of expanding the memory</div><div>on the card.</div><br><blockquote type="cite"><div><blockquote type="cite">(Indeed other threads here have been complaining that 8GB is too little memory).<br></blockquote><br>well, it's not much per-core, especially if, as you suggest elsewhere,<br>it's important to try to use HT (ie, 8G/120 is only 67M/core...)<br></div></blockquote>Agreed, More memory would be great, but... there are space and power issues</div><div>that make that hard at present. (As can be seen by the amount of memory</div><div>on other PCI cards).</div><div><br><blockquote type="cite"><div>I suppose the picture changes if the card can make direct references<br>to host memory, though.<br></div></blockquote></div>Even if it can, I wouldn't recommend it; memory bandwidth is one of the <div>critical issues, and doing cache-line sized fetches over the PCI is guaranteed</div><div>to be horribly slow, so if the PCI is already limiting when you're aggregating the</div><div>transfers into large chunks (by expressing them as offload operations), you can </div><div>expect the performance to be much worse when you try this random access approach.</div><div>(This is similar to one of the reasons that everyone uses MPI, rather than UPC...)</div><div><br></div><div><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">--</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">-- Jim</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">--</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">James Cownie <<a href="mailto:jcownie@cantab.net">jcownie@cantab.net</a>></font></div><br></div></div><br class="Apple-interchange-newline"></div><div><br></div></div></body></html>