Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract colors using color quantization #36

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
edd10cc
Extract colors using color quantization
thekid Nov 4, 2022
2b3609a
Add tests for PriorityQueue
thekid Nov 4, 2022
e94afd4
Fix "Undefined array key 1"
thekid Nov 4, 2022
53cf477
Make quality public
thekid Nov 4, 2022
67a172c
Add histogram() and color()
thekid Nov 4, 2022
5945de8
Use variables for scroll link color in light theme
thekid Nov 5, 2022
10bb183
Merge branch 'main' into feature/colors
thekid Nov 5, 2022
8260bf5
Use generics
thekid Nov 6, 2022
5c2bc15
Use PriorityQueue<Box>
thekid Nov 6, 2022
edce309
Merge branch 'main' into feature/colors
thekid Nov 7, 2022
6a98d78
Merge branch 'main' into feature/colors
thekid Nov 7, 2022
ab842bf
QA: Adjust example to ES6 class refactoring
thekid Nov 8, 2022
c8b0574
Rename color() -> dominant()
thekid Nov 8, 2022
0b03dd5
Rename Colors::DOMINANT_PALETTE -> Colors::DOMINANT
thekid Nov 8, 2022
937f0c4
Pass meta information as JSON (in "meta-inf") or array (in "meta", fo…
thekid Nov 8, 2022
be0baa5
Extract color palette and pass as meta information
thekid Nov 8, 2022
d1fc7fb
Store colors as HSL
thekid Nov 9, 2022
55140d9
First prototypical search by color
thekid Nov 13, 2022
5b81b6d
MFH
thekid Nov 13, 2022
33b50da
MFH
thekid Dec 9, 2022
a86ae0d
Fix missing import statement for Colors
thekid Dec 9, 2022
dacf868
Fix meta extraction
thekid Dec 9, 2022
9b2afe7
Merge branch 'main' into feature/colors
thekid Nov 4, 2023
2eb6a9e
Bump dependency on xp-framework/imaging
thekid Nov 4, 2023
db07a96
Bump dependency on xp-lang/xp-generics
thekid Nov 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
"license": "bsd",

"require": {
"xp-framework/compiler": "^8.5",
"xp-framework/imaging": "^10.1",
"xp-framework/compiler": "^8.6",
"xp-framework/imaging": "^10.2",
"xp-framework/command": "^11.0",
"xp-framework/networking": "^10.4",
"xp-forge/web": "^3.8",
Expand All @@ -17,6 +17,8 @@
"xp-forge/markdown": "^7.0",
"xp-forge/yaml": "^6.0",
"xp-forge/hashing": "^2.1",
"xp-forge/mongodb": "^1.6",
"xp-lang/xp-generics": "^0.7",
"xp-forge/inject": "^5.4",
"xp-forge/mongodb": "^2.0"
},
Expand Down
25 changes: 25 additions & 0 deletions src/main/handlebars/layout.handlebars
Original file line number Diff line number Diff line change
Expand Up @@ -522,13 +522,38 @@
aspect-ratio: auto;
}

.image {
position: relative;
}

.image img, .image video {
aspect-ratio: 15 / 10;
object-fit: cover;
border-radius: .25rem;
box-shadow: .25rem .25rem 1rem rgb(0 0 0 / .2);
}

.image .palette {
position: absolute;
bottom: 1rem;
left: 1rem;
display: flex;
gap: .5rem;
transform: scale(0, 0);
transition: transform ease-in-out 70ms;
}

.image:hover .palette {
transform: scale(1, 1);
}

.image .palette > * {
width: 2rem;
height: 2rem;
border-radius: 50%;
box-shadow: .25rem .25rem 1rem rgb(0 0 0 / .2);
}

#map {
margin-top: 1rem;
width: 100%;
Expand Down
6 changes: 6 additions & 0 deletions src/main/handlebars/partials/images.handlebars
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@
<img alt="{{title}}, {{date meta.dateTime format='d.m.Y H:i'}}" {{#unless first}}loading="lazy"{{/unless}} {{&dataset meta}} src="/image/{{in.slug}}/thumb-{{name}}.webp" width="1024">
</a>
{{/if}}

<div class="palette">
{{#each meta.palette}}
<a href="/search?q=({{h}},{{s}},{{l}})" style="background-color: hsl({{h}} {{s}}% {{l}}%)"> </a>
{{/each}}
</div>
</div>
{{/each}}
</div>
2 changes: 1 addition & 1 deletion src/main/php/de/thekid/dialog/Repository.php
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
class Repository {
private $passwords= Hashing::sha256();

public function __construct(private Database $database) { }
public function __construct(public readonly Database $database) { }

/** Authenticates a given user, returning NULL on failure */
public function authenticate(string $user, Secret $secret): ?Document {
Expand Down
2 changes: 1 addition & 1 deletion src/main/php/de/thekid/dialog/Scripts.php
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
* ```html
* <script type="module">
* {{&use 'suggestions'}}
* suggestions(document.querySelector('#search'));
* const suggestions = new Suggestions('/api/suggestions?q=%s');
* </script>
* ```
*/
Expand Down
4 changes: 3 additions & 1 deletion src/main/php/de/thekid/dialog/api/Entries.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

use de\thekid\dialog\{Repository, Storage};
use io\File;
use text\json\Json;
use util\Date;
use web\rest\{Async, Delete, Entity, Put, Resource, Request, Response, Value};

Expand Down Expand Up @@ -57,6 +58,7 @@ public function upload(#[Value] $user, string $id, string $name, #[Request] $req
foreach ($multipart->files() as $file) {
yield from $file->transmit(new File($f, $file->name()));
}
$meta= $req->param('meta') ?? Json::read($req->param('meta-inf', '{}'));

// Fetch entry again, it might have changed in the meantime!
$images= $this->repository->entry($id, published: false)['images'] ?? [];
Expand All @@ -66,7 +68,7 @@ public function upload(#[Value] $user, string $id, string $name, #[Request] $req
$image= [
'name' => $name,
'modified' => time(),
'meta' => (array)$req->param('meta') + ['dateTime' => gmdate('c')],
'meta' => (array)$meta + ['dateTime' => gmdate('c')],
'is' => [$is => true]
];
foreach ($images as $i => $existing) {
Expand Down
146 changes: 146 additions & 0 deletions src/main/php/de/thekid/dialog/color/Box.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
<?php namespace de\thekid\dialog\color;

/** 3D color space box */
class Box {

private function __construct(private array $components, private Histogram $histogram) { }

/** Creates a box from a given histogram */
public static function from(Histogram $histogram): self {
$min= [256, 256, 256];
$max= [-256, -256, -256];
foreach ($histogram->colors() as $color) {
foreach ($color as $i => $component) {
if ($component < $min[$i]) $min[$i]= $component;
if ($component > $max[$i]) $max[$i]= $component;
}
}
return new self([$min[0], $max[0], $min[1], $max[1], $min[2], $max[2]], $histogram);
}

/** Creates a copy of this box */
public function copy(): self {
return new self($this->components, $this->histogram);
}

/** Volume */
public function volume(): int {
return (
($this->components[1] - $this->components[0] + 1) *
($this->components[3] - $this->components[2] + 1) *
($this->components[5] - $this->components[4] + 1)
);
}

/** Total color count in this box */
public function count(): int {
[$rl, $ru, $gl, $gu, $bl, $bu]= $this->components;

$n= 0;
if ($this->volume() > $this->histogram->size()) {
foreach ($this->histogram->colors() as $count => $c) {
if ($c[0] >= $rl && $c[0] <= $ru && $c[1] >= $gl && $c[1] <= $gu && $c[2] >= $bl && $c[2] <= $bu) {
$n+= $count;
}
}
} else {
for ($r= $rl; $r <= $ru; $r++) {
for ($g= $gl; $g <= $gu; $g++) {
for ($b= $bl; $b <= $bu; $b++) {
$n+= $this->histogram->frequency($r, $g, $b);
}
}
}
}

return $n;
}

/** Returns average color for this box */
public function average(): array<int> {
static $m= 1 << 3;

[$rl, $ru, $gl, $gu, $bl, $bu]= $this->components;
$n= $rs= $gs= $bs= 0;
for ($r= $rl; $r <= $ru; $r++) {
for ($g= $gl; $g <= $gu; $g++) {
for ($b= $bl; $b <= $bu; $b++) {
$h= $this->histogram->frequency($r, $g, $b);
$n+= $h;
$rs+= ($h * ($r + 0.5) * $m);
$gs+= ($h * ($g + 0.5) * $m);
$bs+= ($h * ($b + 0.5) * $m);
}
}
}

if ($n > 0) {
return [(int)($rs / $n), (int)($gs / $n), (int)($bs / $n)];
} else {
return [
min((int)($m * ($rl + $ru + 1) / 2), 255),
min((int)($m * ($gl + $gu + 1) / 2), 255),
min((int)($m * ($bl + $bu + 1) / 2), 255),
];
}
}

/** Returns median boxes or NULL */
public function median(): ?array<self> {
if (1 === $this->count()) return [$this->copy(), null];

// Cut using longest axis
$r= $this->components[1] - $this->components[0];
$g= $this->components[3] - $this->components[2];
$b= $this->components[5] - $this->components[4];

// Rearrange colors
if ($r >= $g && $r >= $b) {
[$c1, $c2]= [0, 1];
[$il, $iu, $jl, $ju, $kl, $ku]= $this->components;
$c= [&$i, &$j, &$k];
} else if ($g >= $r && $g >= $b) {
[$c1, $c2]= [2, 3];
[$jl, $ju, $il, $iu, $kl, $ku]= $this->components;
$c= [&$j, &$i, &$k];
} else {
[$c1, $c2]= [4, 5];
[$jl, $ju, $kl, $ku, $il, $iu]= $this->components;
$c= [&$j, &$k, &$i];
}

$total= 0;
$partial= [];
for ($i= $il; $i <= $iu; $i++) {
$sum= 0;
for ($j= $jl; $j <= $ju; $j++) {
for ($k= $kl; $k <= $ku; $k++) {
$sum+= $this->histogram->frequency(...$c);
}
}
$total+= $sum;
$partial[$i]= $total;
}

for ($i= $il, $h= $total / 2; $i <= $iu; $i++) {
if ($partial[$i] <= $h) continue;

$push= $this->copy();
$add= $this->copy();

// Choose the cut plane
$l= $i - $il;
$r= $iu - $i;
$d= $l <= $r ? min($iu - 1, (int)($i + $r / 2)) : max($il, (int)($i - 1 - $l / 2));

while (empty($partial[$d])) $d++;
while ($partial[$d] >= $total && !empty($partial[$d - 1])) $d--;

$push->components[$c2]= $d;
$add->components[$c1]= $d + 1;
return [$push, $add];
}

return null;
}
}
119 changes: 119 additions & 0 deletions src/main/php/de/thekid/dialog/color/Colors.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<?php namespace de\thekid\dialog\color;

use img\{Color, Image};
use io\File;
use lang\IllegalArgumentException;

/**
* Extracts color palette from a given image
*
* Based on the fabulous work done by Lokesh Dhakar using color quantization
* based on the MMCQ (modified median cut quantization) algorithm from the
* Leptonica library.
*
* @see https://github.com/lokesh/color-thief
* @see https://github.com/olivierlesnicki/quantize
*/
class Colors {
const DOMINANT= 5;
const MAX_ITERATIONS= 1000;
const FRACT_BY_POPULATIONS= 0.75;

/** Creates a new instance with a given quality */
public function __construct(public int $quality= 10) { }

/** Helper for quantize() */
private function iterate(PriorityQueuey<Box> $queue, float $target): void {
$colors= $queue->size();
$iteration= 0;
do {
if ($colors >= $target) return;

$box= $queue->pop();
if (0 === $box->count()) {
$queue->push($box);
continue;
}

[$push, $add]= $box->median();
$queue->push($push);
if ($add) {
$queue->push($add);
$colors++;
}
} while ($iteration++ < self::MAX_ITERATIONS);
}

/**
* Color quantization from a given histogram
*
* @throws lang.IllegalArgumentException if no palette can be computed
*/
private function quantize(Histogram $histogram, int $colors): array<Color> {

// Check border case, e.g. for empty images
if ($histogram->empty()) {
throw new IllegalArgumentException('Cannot quantize using an empty histogram');
}

$queue= new PriorityQueue<Box>();
$queue->push(Box::from($histogram));

// First set of colors, sorted by population
$this->iterate(
$queue->comparing(fn($a, $b) => $a->count() <=> $b->count()),
self::FRACT_BY_POPULATIONS * $colors
);

// Next set - generate the median cuts using the (npix * vol) sorting.
$this->iterate(
$queue->comparing(fn($a, $b) => ($a->count() * $a->volume()) <=> ($b->count() * $b->volume())),
$colors
);

$colors= [];
while ($box= $queue->pop()) {
$colors[]= new Color(...$box->average());
}
return $colors;
}

/**
* Returns histogram for a given image. Uses complete image by default
* but may be given a 4-element array as follows: `[x, y, w, h]`.
*
* The computed histogram for an image can be recycled to derive palette
* and dominant color.
*/
public function histogram(Image $source, ?array $area= null): Histogram {
[$x, $y, $w, $h]= $area ?? [0, 0, ...$source->getDimensions()];

$histogram= new Histogram();
for ($i= 0, $n= $w * $h; $i < $n; $i+= $this->quality) {
$color= $source->colorAt($x + ($i % $w), (int)($y + $i / $w));
$histogram->add($color->red, $color->green, $color->blue);
}
return $histogram;
}

/**
* Returns the dominant color in an image or histogram
*
* @throws lang.IllegalArgumentException if palette is empty
*/
public function dominant(Image|Histogram $source): ?Color {
return current($this->quantize(
$source instanceof Histogram ? $source : $this->histogram($source),
self::DOMINANT
));
}

/**
* Returns a palette with a given size for an image or histogram
*
* @throws lang.IllegalArgumentException if palette is empty
*/
public function palette(Image|Histogram $source, int $size= 10): array<Color> {
return $this->quantize($source instanceof Histogram ? $source : $this->histogram($source), $size);
}
}
Loading
Loading