Hello,

I wanted to ask what solutions are out there use for random accessing BAM files via http.

Of course, the first answer here is samtools/htslib/pysam, but the current version of the htslib creates open range GET requests, those request lead to inflated egress costs when working on the S3 infrastructure.

I described this behavior here:

• https://github.com/samtools/htslib/issues/1670

I was curious, If anybody else experienced this behavior and maybe has an work around for this.

IGV/IGVjs creates clean range requests when accessing data via http, but I don’t see an option to use this functionality outside of the programs for example in a pipeline or a command line tool.

A solution could be to parse the .bai file and define the range for the requested bytes from this data, maybe somebody has some code to share.

Happy about any feedback on this topic.

Best,
Stephan

For command-line work, you could look at using NodeJS with the GMOD bam-js library: https://github.com/GMOD/bam-js

The bam-js library in turn relies on another GMOD library called generic-filehandle that appears to fetch a specified byte range: https://github.com/GMOD/generic-filehandle/blob/06056a2135ddef119262195dd4d5556dfc74b050/src/remoteFile.ts#L72-L134

Perhaps you could run this through a proxy to view headers and confirm if the byte range is calculated more efficiently, or fork the relevant libraries and modify them to write the range header value to a debugger. Or perhaps S3 logs provide enough granularity about requests that would show the same detail of information.

Here's a generic example of a shell script:

#!/usr/bin/env node

const { BamFile } = require('@gmod/bam');
const { performance } = require('perf_hooks');

const bamChrom = 'chr2';
const bamStart = 90383700;
const bamEnd = 90384700;
const bamRootUrl = 'https://foo.cloudfront.net';
const bamUrl = `${bamRootUrl}/reads.bam`;

(async() => {

  const startTime = performance.now();

  const bamHandle = new BamFile({
    bamPath: bamUrl
  });
  const bamHeader = await bamHandle.getHeader();
  const bamReads = await bamHandle.getRecordsForRange(bamChrom, bamStart, bamEnd);
  // console.log(`${JSON.stringify(bamReads)}`);

  const finishTime = performance.now();
  const elapsedTime = finishTime - startTime;
  console.log(`Execution time: ${elapsedTime} ms`);
})();