{"id":11386,"date":"2026-10-04T00:36:25","date_gmt":"2026-10-04T00:36:25","guid":{"rendered":"https:\/\/bitep.net\/blog\/2026\/10\/04\/kalibench-how-a-cybersecurity-tool-use-benchmark-exposes-the-gap-between-agent-i\/"},"modified":"2026-10-04T00:36:25","modified_gmt":"2026-10-04T00:36:25","slug":"kalibench-how-a-cybersecurity-tool-use-benchmark-exposes-the-gap-between-agent-i","status":"publish","type":"post","link":"https:\/\/bitep.net\/blog\/2026\/10\/04\/kalibench-how-a-cybersecurity-tool-use-benchmark-exposes-the-gap-between-agent-i\/","title":{"rendered":"KaliBench: How a Cybersecurity Tool-Use Benchmark Exposes the Gap Between Agent Intent and Executable Commands"},"content":{"rendered":"<h2>Giri\u015f: Kibert\u0259hl\u00fck\u0259sizlik Agentl\u0259rinin \u018fn Z\u0259if B\u0259ndi<\/h2>\n<p>B\u00f6y\u00fck Dil Modell\u0259ri (LLM) \u0259sas\u0131nda f\u0259aliyy\u0259t g\u00f6st\u0259r\u0259n avtonom t\u0259hl\u00fck\u0259sizlik agentl\u0259ri son d\u00f6vrl\u0259rd\u0259 kibert\u0259hl\u00fck\u0259sizlik \u0259m\u0259liyyatlar\u0131n\u0131n m\u0259rk\u0259zin\u0259 \u00e7evrilm\u0259kd\u0259dir. \u0130st\u0259r avtomatla\u015fd\u0131r\u0131lm\u0131\u015f n\u00fcfuzetm\u0259 testl\u0259ri (penetration testing), ist\u0259rs\u0259 d\u0259 insidentl\u0259r\u0259 cavab t\u0259dbirl\u0259ri zaman\u0131 bu agentl\u0259rin q\u0259rarg\u0259bulu y\u00fcks\u0259k qiym\u0259tl\u0259ndirilir. Lakin reall\u0131qda agentl\u0259rin u\u011fursuzluq d\u0259r\u0259c\u0259si h\u0259l\u0259 d\u0259 y\u00fcks\u0259kdir. Bu u\u011fursuzluqlar\u0131n k\u00f6k\u00fcnd\u0259 konseptual bilik \u00e7at\u0131\u015fmazl\u0131\u011f\u0131 deyil, tamamil\u0259 f\u0259rqli bir problem dayan\u0131r: agentin y\u00fcks\u0259k s\u0259viyy\u0259li niyy\u0259tini (intent) d\u0259qiq, i\u015fl\u0259k v\u0259 x\u0259tas\u0131z terminal \u0259mrin\u0259 (CLI invocation) \u00e7evir\u0259 bilm\u0259m\u0259si.<\/p>\n<p>Bu sintaktik v\u0259 funksional bo\u015flu\u011fu \u00f6l\u00e7m\u0259k \u00fc\u00e7\u00fcn haz\u0131rlanm\u0131\u015f <strong>KaliBench<\/strong>, canl\u0131 sisteml\u0259r\u0259 z\u0259r\u0259r vurmadan agentl\u0259rin \u0259mr s\u0259tri al\u0259tl\u0259rind\u0259n istifad\u0259 bacar\u0131\u011f\u0131n\u0131 qiym\u0259tl\u0259ndir\u0259n yeni meyar (benchmark) \u00e7\u0259r\u00e7iv\u0259sidir. Bu m\u0259qal\u0259d\u0259 KaliBench-in i\u015f mexanizmi, agentl\u0259rin n\u0259 \u00fc\u00e7\u00fcn komanda s\u0259trind\u0259 s\u0259hvl\u0259r\u0259 yol verdiyi v\u0259 platforman\u0131n nec\u0259 t\u0259tbiq olunaca\u011f\u0131 d\u0259rind\u0259n ara\u015fd\u0131r\u0131l\u0131r.<\/p>\n<h2>Niyy\u0259t v\u0259 \u0130cra Aras\u0131ndak\u0131 Bo\u015fluq: Problem N\u0259d\u0259dir?<\/h2>\n<p>Bir t\u0259hl\u00fck\u0259sizlik m\u00fct\u0259x\u0259ssisi LLM agentin\u0259 <em>&#8220;H\u0259d\u0259f \u015f\u0259b\u0259k\u0259d\u0259ki 445-ci portu skan et, SMB versiyas\u0131n\u0131 m\u00fc\u0259yy\u0259nl\u0259\u015fdir v\u0259 anonim giri\u015f imkanlar\u0131n\u0131 yoxla&#8221;<\/em> t\u0259limat\u0131n\u0131 verdikd\u0259, model tap\u015f\u0131r\u0131\u011f\u0131n mahiyy\u0259tini tam d\u0259rk edir. Agent h\u0259d\u0259fin SMB servisi oldu\u011funu, hans\u0131 skriptl\u0259rin t\u0259l\u0259b olundu\u011funu v\u0259 risk vektorunu anlay\u0131r. Lakin problem bu niyy\u0259ti <code>nmap<\/code> sintaksisin\u0259 \u00e7evir\u0259rk\u0259n yaran\u0131r.<\/p>\n<p>Komanda S\u0259tri \u0130nterfeysl\u0259ri (CLI) t\u0259bii dild\u0259n f\u0259rqli olaraq he\u00e7 bir sintaktik elastikliy\u0259 d\u00f6z\u00fcml\u00fc deyil. \u018fsas \u00e7\u0259tinlikl\u0259r a\u015fa\u011f\u0131dak\u0131 faktorlardan qaynaqlan\u0131r:<\/p>\n<ul>\n<li><strong>Bayraqlar\u0131n (Flags) D\u0259qiqliyi:<\/strong> T\u0259k tire (<code>-<\/code>) il\u0259 c\u00fct tire (<code>--<\/code>) aras\u0131ndak\u0131 f\u0259rq, b\u00f6y\u00fck-ki\u00e7ik h\u0259rfl\u0259rin h\u0259ssasl\u0131\u011f\u0131 (m\u0259s\u0259l\u0259n, <code>-sV<\/code> \u0259v\u0259zin\u0259 <code>-sv<\/code> yaz\u0131lmas\u0131 \u0259mrin \u00e7\u00f6km\u0259sin\u0259 s\u0259b\u0259b olur).<\/li>\n<li><strong>Arqumentl\u0259rin Ard\u0131c\u0131ll\u0131\u011f\u0131:<\/strong> B\u0259zi yard\u0131m\u00e7\u0131 al\u0259tl\u0259rd\u0259 arqumentl\u0259rin m\u00f6vqeyi (positional arguments) onlar\u0131n i\u015fl\u0259m\u0259 mexanizmini k\u00f6k\u00fcnd\u0259n d\u0259yi\u015fir.<\/li>\n<li><strong>Kombinasiyal\u0131 Parametrl\u0259r:<\/strong> Eyni anda bir ne\u00e7\u0259 modulu i\u015f\u0259 salark\u0259n t\u0259l\u0259b olunan ay\u0131r\u0131c\u0131lar (verg\u00fcll\u0259r, bo\u015fluqlar v\u0259 ya b\u0259rab\u0259rlik i\u015far\u0259l\u0259ri) modell\u0259r t\u0259r\u0259find\u0259n tez-tez qar\u0131\u015fd\u0131r\u0131l\u0131r.<\/li>\n<\/ul>\n<h2>KaliBench N\u0259dir v\u0259 Nec\u0259 \u0130\u015fl\u0259yir?<\/h2>\n<p>KaliBench, Kali Linux ekosistemind\u0259 yer alan populyar t\u0259hl\u00fck\u0259sizlik al\u0259tl\u0259rinin (m\u0259s\u0259l\u0259n, <code>nmap<\/code>, <code>sqlmap<\/code>, <code>gobuster<\/code>, <code>hydra<\/code>, <code>ffuf<\/code> v\u0259 s.) istifad\u0259sini qiym\u0259tl\u0259ndir\u0259n ixtisasla\u015fd\u0131r\u0131lm\u0131\u015f benchmark platformas\u0131d\u0131r. KaliBench-i \u0259n\u0259n\u0259vi qiym\u0259tl\u0259ndirm\u0259 \u00e7\u0259r\u00e7iv\u0259l\u0259rind\u0259n f\u0259rql\u0259ndir\u0259n \u0259n m\u00fch\u00fcm amil onun t\u0259hl\u00fck\u0259sizlik arxitekturas\u0131d\u0131r.<\/p>\n<p>\u018fn\u0259n\u0259vi t\u0259hl\u00fck\u0259sizlik testl\u0259ri agentin \u0259mri icra etm\u0259sini (live execution) t\u0259l\u0259b edir ki, bu da \u015f\u0259b\u0259k\u0259 k\u0259sintil\u0259rin\u0259, sisteml\u0259rin \u00e7\u00f6km\u0259sin\u0259 v\u0259 ya n\u0259zar\u0259tsiz eksploitasiyaya yol a\u00e7a bil\u0259r. KaliBench is\u0259 <strong>t\u0259rc\u00fcm\u0259 qat\u0131n\u0131 (translation layer)<\/strong> t\u0259crid olunmu\u015f v\u0259 deterministik \u015f\u0259kild\u0259 yoxlay\u0131r. Agentin t\u0259qdim etdiyi \u0259mr icra edilmir; \u0259v\u0259zind\u0259 \u0259mrin Abstrakt Sintaksis A\u011fac\u0131 (AST) v\u0259 parametrik t\u0259rkibi m\u00fct\u0259x\u0259ssisl\u0259r t\u0259r\u0259find\u0259n haz\u0131rlanm\u0131\u015f q\u0131z\u0131l standartlarla (ground truth) m\u00fcqayis\u0259 edilir.<\/p>\n<h3>\u018fsas G\u00f6st\u0259ricil\u0259r v\u0259 M\u00fcqayis\u0259<\/h3>\n<p>A\u015fa\u011f\u0131dak\u0131 c\u0259dv\u0259ld\u0259 canl\u0131 m\u00fchit testl\u0259ri il\u0259 KaliBench yana\u015fmas\u0131n\u0131n texniki f\u0259rql\u0259ri \u00fcmumil\u0259\u015fdirilmi\u015fdir:<\/p>\n<table>\n<thead>\n<tr>\n<th>X\u00fcsusiyy\u0259t<\/th>\n<th>Canl\u0131 \u0130cra Meyarlar\u0131 (Live Execution)<\/th>\n<th>KaliBench Yana\u015fmas\u0131<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>T\u0259hl\u00fck\u0259sizlik Riski<\/strong><\/td>\n<td>Y\u00fcks\u0259k (Sisteml\u0259r\u0259 z\u0259r\u0259r vurma riski)<\/td>\n<td>S\u0131f\u0131r (He\u00e7 bir da\u011f\u0131d\u0131c\u0131 \u0259mr icra olunmur)<\/td>\n<\/tr>\n<tr>\n<td><strong>T\u0259krarlanma D\u0259r\u0259c\u0259si<\/strong><\/td>\n<td>A\u015fa\u011f\u0131 (\u015e\u0259b\u0259k\u0259 gecikm\u0259l\u0259ri v\u0259 v\u0259ziyy\u0259t d\u0259yi\u015fiklikl\u0259ri)<\/td>\n<td>Y\u00fcks\u0259k (100% Deterministik n\u0259tic\u0259l\u0259r)<\/td>\n<\/tr>\n<tr>\n<td><strong>Hesablama X\u0259rci<\/strong><\/td>\n<td>Y\u00fcks\u0259k (H\u0259d\u0259f virtual ma\u015f\u0131nlar t\u0259l\u0259b olunur)<\/td>\n<td>A\u015fa\u011f\u0131 (Yaln\u0131z sintaksis v\u0259 struktur analizi)<\/td>\n<\/tr>\n<tr>\n<td><strong>X\u0259tan\u0131n Lokalla\u015fd\u0131r\u0131lmas\u0131<\/strong><\/td>\n<td>\u00c7\u0259tin (\u018fmrin niy\u0259 u\u011fursuz oldu\u011fu ayd\u0131n deyil)<\/td>\n<td>D\u0259qiq (S\u0259hv bayraq v\u0259 parametr d\u0259rhal a\u015fkarlan\u0131r)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Add\u0131m-add\u0131m T\u0259limat: KaliBench-in Qura\u015fd\u0131r\u0131lmas\u0131 v\u0259 \u0130stifad\u0259si<\/h2>\n<p>KaliBench platformas\u0131n\u0131 lokal m\u00fchitinizd\u0259 i\u015f\u0259 salaraq t\u0259hl\u00fck\u0259sizlik y\u00f6n\u00fcml\u00fc LLM-l\u0259rinizin v\u0259 ya avtonom agentl\u0259rinizin performans\u0131n\u0131 birba\u015fa yoxlaya bil\u0259rsiniz.<\/p>\n<h3>1. T\u0259l\u0259bl\u0259r v\u0259 Repozitoriyan\u0131n \u018fld\u0259 Edilm\u0259si<\/h3>\n<p>\u0130lk olaraq sisteminizd\u0259 Python 3.10 v\u0259 ya daha y\u00fcks\u0259k versiyan\u0131n qura\u015fd\u0131r\u0131ld\u0131\u011f\u0131ndan \u0259min olun. Layih\u0259ni r\u0259smi repozitoriyadan klonlay\u0131n v\u0259 virtual m\u00fchit yarad\u0131n:<\/p>\n<pre><code class=\"language-bash\">git clone https:\/\/github.com\/your-org\/kalibench.git\ncd kalibench\npython3 -m venv venv\nsource venv\/bin\/activate\npip install --upgrade pip\npip install -r requirements.txt<\/code><\/pre>\n<h3>2. \u018ftraf M\u00fchit D\u0259yi\u015f\u0259nl\u0259rinin T\u0259nziml\u0259nm\u0259si<\/h3>\n<p>Test ed\u0259c\u0259yiniz LLM provayderinin a\u00e7arlar\u0131n\u0131 (m\u0259s\u0259l\u0259n, OpenAI, Anthropic v\u0259 ya lokal vLLM\/Ollama endpoint-l\u0259rini) konfiqurasiya edin:<\/p>\n<pre><code class=\"language-bash\">cp .env.example .env\nexport OPENAI_API_KEY=\"sk-...\"\nexport ANTHROPIC_API_KEY=\"claude-...\"<\/code><\/pre>\n<h3>3. Qiym\u0259tl\u0259ndirm\u0259 Testl\u0259rinin \u0130\u015f\u0259 Sal\u0131nmas\u0131<\/h3>\n<p>A\u015fa\u011f\u0131dak\u0131 \u0259mrl\u0259 m\u00fc\u0259yy\u0259n bir al\u0259t qrupu (m\u0259s\u0259l\u0259n, \u015f\u0259b\u0259k\u0259 k\u0259\u015ffiyyat\u0131 v\u0259 ya veb z\u0259iflikl\u0259ri) \u00fczr\u0259 agentin komanda generasiya d\u0259qiqliyini qiym\u0259tl\u0259ndirin:<\/p>\n<pre><code class=\"language-bash\"># Nmap v\u0259 \u015f\u0259b\u0259k\u0259 skan al\u0259tl\u0259ri \u00fczr\u0259 qiym\u0259tl\u0259ndirm\u0259\npython -m kalibench.cli evaluate \\\n    --model gpt-4o \\\n    --category network_recon \\\n    --output results\/network_results.json<\/code><\/pre>\n<h3>4. N\u0259tic\u0259l\u0259rin T\u0259hlili<\/h3>\n<p>\u0130cra tamamland\u0131qdan sonra KaliBench \u0259trafl\u0131 statistik hesabat t\u0259qdim edir. Hesabatda sintaktik uy\u011funluq (Syntactic Validity), bayraq d\u0259qiqliyi (Flag Recall &amp; Precision) v\u0259 arqument s\u0131ralamas\u0131 faizl\u0259rl\u0259 g\u00f6st\u0259rilir:<\/p>\n<pre><code class=\"language-bash\">python -m kalibench.cli report --file results\/network_results.json --format summary<\/code><\/pre>\n<h2>N\u0259tic\u0259 v\u0259 S\u0259nayey\u0259 T\u0259siri<\/h2>\n<p>KaliBench s\u00fcbut edir ki, avtonom agentl\u0259rin inki\u015faf\u0131nda \u0259n b\u00f6y\u00fck mane\u0259 t\u0259hl\u00fck\u0259sizlik ontologiyas\u0131n\u0131 anlamaq deyil, deterministik sintaksis generasiyas\u0131d\u0131r. Agentl\u0259rin &#8220;d\u00fc\u015f\u00fcnc\u0259 t\u0259rzi&#8221; (reasoning) il\u0259 komanda s\u0259tri aras\u0131ndak\u0131 bu bo\u015fluq qapad\u0131lmad\u0131qca, avtonom Red Teaming v\u0259 ya Blue Team agentl\u0259rini kritik istehsalat m\u00fchitl\u0259rin\u0259 buraxmaq ciddi \u0259m\u0259liyyat riskl\u0259ri yaradacaqd\u0131r.<\/p>\n<p>G\u0259l\u0259c\u0259kd\u0259 al\u0259tl\u0259rd\u0259n istifad\u0259 ed\u0259n (tool-augmented) modell\u0259rin t\u0259lim prosesind\u0259 KaliBench kimi deterministik m\u00fchitl\u0259rd\u0259n istifad\u0259 edilm\u0259si, agentl\u0259rin sintaksis x\u0259talar\u0131n\u0131 minimuma endir\u0259c\u0259k v\u0259 kibert\u0259hl\u00fck\u0259sizlik \u0259m\u0259liyyatlar\u0131n\u0131n avtomatla\u015fd\u0131r\u0131lmas\u0131n\u0131 yeni m\u0259rh\u0259l\u0259y\u0259 da\u015f\u0131yacaqd\u0131r.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Giri\u015f: Kibert\u0259hl\u00fck\u0259sizlik Agentl\u0259rinin \u018fn Z\u0259if B\u0259ndi B\u00f6y\u00fck Dil Modell\u0259ri (LLM) \u0259sas\u0131nda f\u0259aliyy\u0259t g\u00f6st\u0259r\u0259n avtonom t\u0259hl\u00fck\u0259sizlik agentl\u0259ri son d\u00f6vrl\u0259rd\u0259 kibert\u0259hl\u00fck\u0259sizlik \u0259m\u0259liyyatlar\u0131n\u0131n m\u0259rk\u0259zin\u0259 \u00e7evrilm\u0259kd\u0259dir. \u0130st\u0259r avtomatla\u015fd\u0131r\u0131lm\u0131\u015f n\u00fcfuzetm\u0259 testl\u0259ri (penetration testing), ist\u0259rs\u0259 d\u0259 insidentl\u0259r\u0259 cavab t\u0259dbirl\u0259ri zaman\u0131 bu agentl\u0259rin q\u0259rarg\u0259bulu y\u00fcks\u0259k qiym\u0259tl\u0259ndirilir. Lakin reall\u0131qda agentl\u0259rin u\u011fursuzluq d\u0259r\u0259c\u0259si h\u0259l\u0259 d\u0259 y\u00fcks\u0259kdir. Bu u\u011fursuzluqlar\u0131n k\u00f6k\u00fcnd\u0259 konseptual bilik \u00e7at\u0131\u015fmazl\u0131\u011f\u0131 deyil, tamamil\u0259 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":11385,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11386","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/bitep.net\/blog\/wp-content\/uploads\/2026\/10\/bitep-ai-1791074179915-15.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/posts\/11386","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/comments?post=11386"}],"version-history":[{"count":0,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/posts\/11386\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/media\/11385"}],"wp:attachment":[{"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/media?parent=11386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/categories?post=11386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bitep.net\/blog\/wp-json\/wp\/v2\/tags?post=11386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}