<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>TextLap: Customizing Language Models for Text-to-Layout Planning</title></titleStmt>
			<publicationStmt>
				<publisher>The 2024 Conference on Empirical Methods in Natural Language Processing</publisher>
				<date>11/12/2024</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10637261</idno>
					<idno type="doi"></idno>
					
					<author>Jian Chen</author><author>Ruiyi Zhang</author><author>Yufan Zhou</author><author>Jennifer Healey</author><author>Jiuxiang Gu</author><author>Zhiqiang Xu</author><author>Changyou Chen</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces. Given the incredible ability of Large language models (LLMs) in both natural language understanding and generation, we believe that we could customize an LLM to help people create compelling graphical layouts starting with only text instructions from the user. We call our method TextLap (text-based layout planning) 1 . It uses a curated instruction-based layout planning dataset (InsLap) to customize LLMs as a graphic designer. We demonstrate the effectiveness of TextLap and show that it outperforms strong baselines, including GPT-4 based methods, for image generation and graphical design benchmarks.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Traditional layout generation tasks typically involve organizing text and graphical elements into a pleasing and professional looking arrangement. This is often time-consuming and difficult for users, especially those without training or design skill. Our goal is to condition an LLM so that it allows users to generate professional looking layouts by simply inputting text instructions. We are inspired by the rapid advancement of generative models for text <ref type="bibr">(Ouyang et al., 2022;</ref><ref type="bibr">Achiam et al., 2023)</ref>, images <ref type="bibr">(Betker et al., 2023;</ref><ref type="bibr">Podell et al., 2023)</ref>, and videos <ref type="bibr">(Peebles and Xie, 2023)</ref> and recent works that create layouts using graphical elements such as category, position, or size as input <ref type="bibr">(Li et al., 2020;</ref><ref type="bibr">Kikuchi et al., 2021;</ref><ref type="bibr">Jyothi et al., 2019;</ref><ref type="bibr">Inoue et al., 2023)</ref>. We advance the field by customizing LLMs as text-guided layout generation models that require only a text description of the desired layout and optionally text descriptions for visual elements as input, enabling layout planning solely in the text modality. We believe that building an LLM-based layout generation offers a more user-friendly approach to achieving desired designs, allowing text instructions to guide the process-an aspect that is challenging to incorporate in traditional settings.</p><p>Creating 2D graphical layouts from text alone is challenging for several reasons. First, text descriptions of 2D arrangements are usually underspecified. There are multiple ways to describe a 2D layout, and there are multiple 2D layouts that can meet a description. Furthermore, comprehending 2D spatial relations <ref type="bibr">(Xu et al., 2020)</ref> is challenging for LLMs due to their autoregressive nature and 1D positional embedding <ref type="bibr">(Vaswani et al., 2017)</ref>. Even understanding the meaning of coordinate numbers in a description is sometimes difficult for LLMs.</p><p>To enable the generation of 2D designs from text, we build an instruction-based layout planning (In-stLap) dataset to customize large language models. We then use InstLap to train our Text-based Layout Planning (TextLap) model through supervised fine-tuning of an LLM, enabling it to understand 2D spatial relationships and generate or modify bounding box coordinates based on text prompts. Figure <ref type="figure">1</ref> illustrates example inputs and outputs of the proposed TextLap model.</p><p>As an LLM, TextLap enables users to iteratively refine layout designs through natural language conversations. Initial text instructions provide an initial draft, and users can further customize their designs by interacting with the model. TextLap generates text-coherent layouts in response to users' iterative requests, enabling them to quickly find the most suitable custom template for their design. We believe that TextLap significantly reduces the time required to create designs, enhances design efficiency, and plays an important role in assisting TextLap, and its empirical evaluations on benchmark datasets show its superior performance compared to GPT-4 models.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Related Work</head><p>Layout Generation Models Various approaches have been developed for controllable layout generation. <ref type="bibr">Jiang et al. (2023)</ref>; <ref type="bibr">Arroyo et al. (2021)</ref>; <ref type="bibr">Gupta et al. (2021)</ref> utilize transformers to create auto-regressive models that produce layouts as a sequence of element attributes. Adversarial generative models <ref type="bibr">(Kikuchi et al., 2021;</ref><ref type="bibr">Li et al., 2020)</ref> and diffusion models <ref type="bibr">(Chen et al., 2024;</ref><ref type="bibr">Inoue et al., 2023;</ref><ref type="bibr">Zhang et al., 2023)</ref> have been used to enhance the quality of generation and unify various conditional generation tasks in a single model. TextDiffuser <ref type="bibr">(Chen et al., 2023b)</ref> applies a layout transformer to generate layouts for keywords in text-rich images. <ref type="bibr">(Jin et al., 2022;</ref><ref type="bibr">Hsu et al., 2023;</ref><ref type="bibr">Yu et al., 2022)</ref> generate background-aware layouts for posters by detecting smooth regions of the background image. <ref type="bibr">(Yang et al., 2022)</ref> introduces a graph transformer architecture to generate layouts for images based on a scene graph that describes the relationship between visual objects. The visual layout serves as a crucial spatial control in image generation. GLIGEN <ref type="bibr">(Li et al., 2023)</ref> integrates gated self-attention to enable spatial grounding capabilities in pre-trained diffusion models. Similarly, InstanceDiffusion <ref type="bibr">(Wang et al., 2024)</ref> integrates locations and descriptions at the instance level into the generation process. TextLap is an LLM-based layout planning model that can be used to design various items, such as posters and book covers. LLM-based Layout Generation Recent studies have introduced approaches based on language models to enable interactive layout generation and modification through chatting. TextDiffuser-2 <ref type="bibr">(Chen et al., 2023a)</ref> fine-tunes a large language model to generate keywords for text rendering. <ref type="bibr">(Yang et al., 2024)</ref> demonstrate that the integration of programming code in LLM training benefits the performance of LLM agents in various tasks. LayoutNUWA <ref type="bibr">(Tang et al., 2023)</ref> leverages a large language model to generate layouts as HTML code.</p><p>LayoutGPT <ref type="bibr">(Feng et al., 2023)</ref> in-context visual demonstrations in CSS structures to enhance the visual planning skills of GPT-3.5/4 <ref type="bibr">(Ouyang et al., 2022;</ref><ref type="bibr">Achiam et al., 2023)</ref> to plan layouts from text conditions. MuLan <ref type="bibr">(Li et al., 2024)</ref> iteratively plans the layout of an image by breaking down the text prompt into a sequence of sub-tasks with an LLM, and then revises the image at each step based on feedback from a vision-language model (VLM).</p><p>InstructScene <ref type="bibr">(Lin and Mu, 2024)</ref> uses ChatGPT to filter the caption describing the layout elements. TextLap sets itself apart from prior research by delving into the text-to-layout task, which allows users to generate a layout design based on naturallanguage descriptions. 2 &#169;2020 Adobe. All Rights Reserved. Adobe Confid Instructions: A potted plant is to the left of a tv, which is at the center; a laptop is on the lower right. COCO Image + Caption + Bbox LLM-based Augmentation: A bonsai in a ceramic pot is to the left of a flat screen LED TV, which is at the center; a sleek silver laptop is on the lower right.. COCO Bbox (Target Output): potted plant: [ 0, 20, 30, 104] Tv: [43, 49, 77, 75] laptop: [74, 103, 104, 111] Shift the laptop to the right by 23 Shift Augmentation: potted plant: [ 0, 20, 30, 104] Tv: [43, 49, 77, 75] laptop: [97, 103, 127, 111] GPT-4 GPT-4V GPT-4V SVGs Images Texts Instructions: On a canvas with a width of 940 and a height of 788, A list of visual elements with their descriptions is given in CSS format: [ele_1: {description}, ele_2:{}, &#8230;, ele_n: {}]. Please generate the bounding box coordinates (x_min, y_min, x_max, y_max) in CSS format. MS Phi-3-Vision Coordinates (Target Output): background {left: 0px; top: 0px; width: 940px; height: 788px; } imageElement1 { left: 91px; top: 9px; width: 772px; height: 772px; } textElement1 { left: 327px; top: 716px; width: 294px; height: 26px; }. Crello Dataset: Elements + Bbox Heuristic Filters Shift Augmentation: textElement1 { left: 350px; top: 716px; width: 294px; height: 26px; } </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Creating the InstLap Dataset</head><p>To customize a language model for layout planning, we developed the Instruction to Layout Planning (InstLap) dataset which comprises filtered and enhanced image-caption pairs from MS-COCO (2017) <ref type="bibr">(Lin et al., 2014)</ref>, a subset of the LAION datset <ref type="bibr">(Schuhmann et al., 2022;</ref><ref type="bibr">Chen et al., 2023b)</ref>, and Crelllo <ref type="bibr">(Yamaguchi, 2021)</ref>.</p><p>Table 1 provides the statistics for the InstLap dataset, which includes human-annotated benchmarks, the original dataset, and augmented samples. Domain Task Train Test Bench Image Layout Planning 27,246 1,255 -Keywords Aug. 22,364 1,039 -Layout Shift 22,364 --Text Layout Planning 4,873 -502 Text Split 4,873 --Graphic Layout Planning 16119 1954 -Layout Shift 16176 --Table 1: Number of examples in each training and testing of the InstructLap datasets. Tasks refer to the original layout planning data processed and its different augmentation methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">MS COCO Layout</head><p>We conducted data processing and filtering to remove abnormal samples, resulting in 27,246 textto-layout pairs for layout planning tasks. We then generated textual descriptions for these layouts by prompting GPT-4V. we use different augmentation methods to obtain more instruction tuning data and split the dataset into training and testing for visual layout planning task.</p><p>Layout Normalization We standardized the size of the images in MS COCO by scaling the longest axis to a uniform length and centering it within a square canvas. This adjustment also involved recalibrating the bounding box coordinates to ensure that the scaling process did not distort the appearance of objects.</p><p>Object Selection In the annotations, small bounding boxes were discarded to simplify the layout. An area threshold &#969; a = 0.1 was applied to the largest box in an image to avoid layouts cluttered with numerous small objects. Furthermore, a threshold &#969; l = 0.2 was established for the maximum dimension (height or width) of all boxes in each image to remove background objects that are too small or irrelevant to the overall scene. These thresholds were manually selected.</p><p>Overlapping Filtering Overlapping bounding boxes were considered undesirable as they could introduce conflicting guidance in the same region, potentially leading to localized inaccuracies or "hallucinations" in the generated image. To address this issue, we calculated a pairwise matrix of Intersection over Union (IoU) scores for all bounding boxes within each image. A threshold &#969; o = 0.01 was selected as the maximum value to filter out images with significant overlaps of the boxes.</p><p>Crowd Filtering Images in MS COCO with the 'is_crowd' label in their annotations were removed. This label indicates cases where a bounding box encompasses multiple instances of the same class under a single label (e.g., a densely populated area of people annotated with the singular label "person"). Such cases can potentially make LLMs confused when estimating the size of an individual.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Graphic Design Data</head><p>We also built InstLap based on Crello <ref type="bibr">(Yamaguchi, 2021)</ref> for automatic graphic designs. Specifically, we first merge three types of elements into the background: i) tiny elements that occupy less than 1% of the canvas and ii) elements with more than 70% transparent pixels, which means that their shapes 3 are difficult to describe as bounding boxes. We then filter the dataset based on the number of remaining elements, discarding those with more than 10 elements resulting in 16,119 training layouts and 1,954 layouts for testing.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">LLM-Assisted Annotation</head><p>Layout Prompt Generation We observed a significant inconsistency between object detection and caption annotations within the MS COCO dataset.</p><p>Frequently, many objects either lack references in any of the five captions or are described using different terminology. For example, an object labeled as "person" in the annotation might be referred to as "woman" in the caption. As a result, caption annotations cannot reliably serve as prompts for layout generation. A detailed example is given in Appendix A.</p><p>To address this challenge, we capitalize on the observational and interpretative capabilities of large vision-language models, such as GPT-4V <ref type="bibr">(Yang et al., 2023)</ref>, LLaVA <ref type="bibr">(Liu et al., 2023)</ref>, and Phi-3-Vision <ref type="bibr">(Abdin et al., 2024)</ref>. We utilize these models to generate text prompts that describe given layouts, which consist of selected objects based on box annotations and the original image. To ensure consistency between the generated prompt and layout, we provide manually crafted examples to illustrate desired captions for specific layouts. Additionally, we instruct the model not to mention any objects in the image that are not included among the selected objects.</p><p>In the Crello dataset, the elements are too varied to be categorized or described using a predetermined set of labels. Consequently, we utilized the Phi-3-Vision model to create a brief description for every element in the dataset, including the background for content-aware graphic designs. In addition, we employed a heuristic method to create spatial descriptions for major elements in the design. All element captions, sizes, and spatial instructions are added in the prompt, providing contexts for LLM when generating vibrant layouts.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Object Augmentation</head><p>The MS COCO dataset employs a set of labels of 80 classes, which would restrict the generalizability of LLM models. We utilize an LLM to augment the labels to facilitate open-set layout inference. Specifically, we provide carefully curated examples as context for GPT-4 <ref type="bibr">(Achiam et al., 2023)</ref> and ask it to replace the single word tag of a box with a phrase that describes an object of comparable size and similar semantics, ensuring that the enhancement remains concise and natural.</p><p>Layout Shift Augmentation We enrich the training data with instruction-following pairs to promote the model's understanding of the relationship between bounding-box coordinates and directions. We guide the model in moving an object in a specified direction for a random distance. To prevent overlaps, we ask the model to move objects at the boundary further from adjacent objects, e.g., moving the leftmost object further to the left. This strategy boosts the model's spatial awareness and ensures the integrity of the data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.4">Visual Text Layout Data</head><p>We have curated a collection of 4,873 layout prompts specifically tailored for visual text analysis. These prompts were generated from TextDiffuser-2 training data samples <ref type="bibr">(Chen et al., 2023a)</ref>, which include caption-OCR pairs derived from the MARIO-10M dataset. We developed two distinct prompts for each layout to improve TextLap's ability to adhere to user instructions for keyword splitting and to autonomously handle keyword segmentation. One prompt explicitly demonstrates how keywords are split, providing clear guidance for the process. The other prompt delegates the segmentation task to TextLap, allowing the model to autonomously determine the optimal segmentation approach based on the layout context.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>InstaLap Bench</head><p>We build InstLap Bench following the TRINS-Gen dataset <ref type="bibr">(Zhang et al., 2024b)</ref>. Considering the difficulty of rendering too many words in a single image, we run Paddle OCR and filter out images with more than 10 OCR words, resulting in a curated set of 502 images. We hired annotators from Upwork to write human annotations for each given image, which describe both textual and visual objects. Each annotator has to explicitly describe words and their positions within the image. To this end, we introduce InstLap-Bench, the visually-rich document design benchmark meticulously annotated by humans.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Model Training</head><p>We construct the TextLap model by fine-tuning the Vicuna-1.5 7B <ref type="bibr">(Chiang et al., 2023)</ref> model using the FastChat framework <ref type="bibr">(Zheng et al., 2024)</ref>. The model is trained on the InstLap-train set with both 4 object layouts and text layouts. Figure <ref type="figure">2</ref> presents examples of conversations from the InstLap-train dataset. We train the model using both the closedset layout prompt and the prompt enhanced with object augmentations. The training aims to enable the model to extract objects and separate visualtext elements when no specific target is provided. Additionally, the model is trained to focus on userspecified keywords or modification instructions. Training details are included in Appendix B.1.</p><p>We use multiple text formats for layout representation, including lists of integer/float numbers and CSS structures. However, we notice that multidigit coordinates are often tokenized into multiple tokens, potentially complicating the model's numerical understanding. We integrated discrete coordinates as single tokens into the tokenizer to mitigate this.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Experimental Results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">Experiments Setup</head><p>We evaluate our model's capability to generate layouts for objects and visual text using the InstLap-Test split. All TextLap varaints utilize the same set of weights trained on the InstLap-Train set. We first assess the model's performance in close-set object layout generation, which feature a defined label set of 80 classes, and an open-set scenario that include LLM-augmented object descriptions.</p><p>In both experiments, we present prompts with and without specific objects to evaluate the TextLap on identifying key objects within the text prompts.</p><p>Then, we conduct an experiment using the Crello dataset <ref type="bibr">(Yamaguchi, 2021)</ref> for graphic design given a list of elements, including images, texts, and SVGs.</p><p>In addition, we evaluate our model for identifying visual-text elements and creating visual-text layouts using the InstLap-Bench split. The prompts in this dataset are significantly more complex than the 5k visual-text training samples in the InstLap-Train set. All experiments are implemented with PyTorch and performed on Nvidia A100 GPUs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Visual Layout Generation</head><p>While image generation has greatly benefited from the continual expansion of diffusion models, controllability remains a challenge, particularly in scenarios involving multiple user-imposed conditions or constraints. Current generative models like DALL&#8226;E 3 <ref type="bibr">(Betker et al., 2023)</ref> and SDXL <ref type="bibr">(Podell et al., 2023)</ref> still struggle to generate samples that satisfy all conditions. These challenges underscore the limitations of relying solely on larger models, prompting the need to incorporate additional layout planning components to better address the problem.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Evaluation Metrics</head><p>To assess the quality of the generated content, we employ six computational metrics: (1) Fr&#233;chet Inception Distance (FID) <ref type="bibr">(Heusel et al., 2017)</ref>: Assesses the similarity between distributions of generated layouts and validated layouts, capturing both the variety and fidelity of the generated layouts. We modified the network structure to adapt the open-set layout. The model is introduced in Appendix B.2. ( <ref type="formula">2</ref>) MaxIoU (Maximum Intersection-over-Union): this metric assesses the overlap between the generated and target layouts by calculating the average IoU of the optimally matched element pairs. We adapt the original definition by <ref type="bibr">(Kikuchi et al., 2021)</ref>, focusing on matching layouts generated from the same prompt rather than from the same set of labels. For object layouts, we find the best-matched box among the boxes that share the same label. Finding exact matches is challenging for visual-text layouts, where box labels are segments of phrases that may be split differently. Therefore, we define the closest match using the cosine similarity of CLIP text features. (3) Failure rate: the proportion of layouts generated by the LLM in an invalid format that cannot be processed automatically. (4) Precision: Accuracy of correctly identified elements within all extracted elements. (5) Recall: The ratio of correctly identified elements within all ground truth elements. ( <ref type="formula">6</ref>) F-score: The harmonic mean of Precision and Recall, balancing their contributions.</p><p>Settings To evaluate the effectiveness of text-tolayout models, we use text-to-image generation as a downstream task. We compare our method, TextLap, with LayoutGPT <ref type="bibr">(Feng et al., 2023)</ref> and GPT-4-based baselines for close-set and open-set text-guided layout generation. The naive baseline is termed GPT-4, where we ask GPT-4 to generate a layout for a given prompt with three fixed examples to demonstrate the desired output format. We introduce two additional baselines, GPT-4 (R) and GPT-4 (rCSS), for our experiments designed to enhance in-context learning for the GPT-4 model. These baselines employ cosine similarity of CLIP text features to identify and retrieve the most relevant demonstration exam-</p><p>5 Methods Text prompts with target objects FID &#8595; MaxIoU &#8593; Fail % &#8595; F-score &#8593; GPT-4 382.0 0.292 0.956 0.989 GPT-4 (R) 26.40 0.452 1.116 0.979 GPT-4 (rCSS) 37.45 0.459 1.116 0.969 LayoutGPT 248.0 0.435 0.000 0.959 TextLap-S128 18.64 0.454 2.151 0.974 TextLap-D128 13.54 0.475 0.398 0.983 TextLap-D1024 14.43 0.456 0.398 0.973 TextLap-Float 15.09 0.475 0.159 0.996 TextLap-CSS 19.32 0.458 0.000 0.998 Text prompts only GPT-4 504.0 0.005 98.566 0.012 GPT-4 (R) 30.01 0.417 4.382 0.856 GPT-4 (rCSS) 39.17 0.427 1.594 0.867 LayoutGPT 265.7 0.416 0.000 0.900 TextLap-S128 20.67 0.452 1.116 0.976 TextLap-D128 14.77 0.267 25.976 0.537 TextLap-D1024 16.27 0.347 9.163 0.716 TextLap-Float 14.36 0.424 3.426 0.877 TextLap-CSS 18.69 0.440 0.080 0.979 Table 2: Comparative results of close-set layout generation with 80-class COCO labels Methods Text prompts with target objects FID &#8595; MaxIoU &#8593; Fail % &#8595; F-score &#8593; GPT-4 291.8 0.330 1.347 0.979 GPT-4 (R) 48.59 0.421 1.925 0.974 GPT-4 (rCSS) 41.73 0.456 0.481 0.972 LayoutGPT 261.4 0.383 0.000 0.824 TextLap-D128 17.77 0.485 1.059 0.972 TextLap-D1024 21.18 0.467 0.385 0.949 TextLap-Float 18.16 0.456 0.481 0.937 TextLap-CSS 19.48 0.464 0.000 0.996 Text prompts only GPT-4 398.4 0.006 95.091 0.011 GPT-4 (R) 78.03 0.155 9.047 0.315 GPT-4 (rCSS) 88.66 0.093 2.406 0.197 LayoutGPT 269.4 0.326 0.000 0.712 TextLap-D128 16.38 0.322 18.383 0.642 TextLap-D1024 18.34 0.349 10.298 0.714 TextLap-Float 16.67 0.309 3.272 0.633 TextLap-CSS 19.44 0.446 0.192 0.960</p><p>Table 3: Results on open-set layout generation with LLM-augmented labels</p><p>ples from the InstLap-Train dataset for each test prompt <ref type="bibr">(Feng et al., 2023)</ref>. They differ in their approach to representing layout coordinates: GPT-4 (R) uses lists of integers, while GPT-4 (rCSS) adopts a CSS-like structure with a maximum value of 128. We finetuned four TextLap variants with different coordinate representations: discrete coordinates on 128!128 and 1024!1024 canvases (TextLap-D128, TextLap-D1024), floating-point coordinates (TextLap-Float), and a CSS format with a max value of 128 (TextLap-CSS). Additionally, we trained a model with 128 new tokens for discrete coordinates, called TextLap-S128.</p><p>RQ1: Does simply scale up the LLM help for design problems? Table <ref type="table">2</ref> presents the results of TextLap models alongside three GPT-4-based methods. In particular, TextLap models achieve significantly lower layout FID scores compared to GPT-4 baselines, highlighting the advantages of fine-tuning. In the open-set generation results, presented in Table <ref type="table">3</ref>, there is a notable decrease in the retrieval scores for GPT-4 baselines when objects are not specified in the prompt. This decline could be attributed to the fact that, in close-set experiments, the retrieval process often identifies examples with matching objects, implicitly providing the necessary keywords in context. This shows that task-specific models can potentially outperform larger models and simple scaling-up does not help for design problems.</p><p>RQ2: What is the impact of different layout representations? Among various TextLap variants, TextLap-CSS stands out, outperforming all baselines by a considerable margin and demonstrating high stability in object extraction. TextLap-S128 showed inferior performance, likely due to inadequate training data to effectively integrate new weights. GPT-4 (rCSS) also exhibits the lowest failure rates among the baseline methods, consistent with insights from previous studies by <ref type="bibr">(Yang et al., 2024;</ref><ref type="bibr">Feng et al., 2023;</ref><ref type="bibr">Tang et al., 2023)</ref>, which suggest that pre-trained large language models better understand programming patterns, possibly due to exposure to code snippets in their training data.</p><p>RQ3: Are special tokens of coordinates useful? TextLap-S128, which incorporates special coordinate tokens, performs well on closed-set data, with a lower failure rate in text prompt-only settings compared to TextLap-D128 and TextLap-Float. However, it struggles in open-set generation, likely due to the large number of new parameters requiring more training data and time. As shown in the loss curve in Figure <ref type="figure">3</ref>, TextLap-S128 had higher loss after the same number of epochs and converged to a slightly higher loss even with extended training, indicating a need for more training to achieve similar performance. Fine-tuning existing tokens, which are well-trained in LLMs and handle arithmetic effectively, proves more efficient. Thus, adding special tokens is less effective than a CSS-based layout representation, especially for open-set tasks, where large-scale pre-training may help <ref type="bibr">(Lv et al., 2023)</ref>.</p><p>6 </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">Visual Text Layout Generation</head><p>We evaluate the ability of our models in generating layouts for visual-text using the InstLap-Bench set. Specifically, we compare TextLap-Float and TextLap-CSS with GPT-4 and GPT-4 (rCSS). Additionally, we benchmark our models against the large language model (LLM) employed in TextD-iffuser2, which is fine-tuned on the same 5,000 visual-text layout training samples, utilizing the Vicuna-v1.5 7B checkpoint.</p><p>Methods MaxIoU (suc) &#8593; MaxIoU &#8593; Fail % &#8595; GPT-4 0.231 0.206 10.778 GPT-4 (rCSS) 0.252 0.241 4.192 TextDiffuser-2 0.166 0.165 0.80 TextLap-Float 0.209 0.096 54.09 TextLap-CSS 0.211 0.211 0.00</p><p>Table 4: Results on text layout generation on the InstLap-Bench.</p><p>Table <ref type="table">4</ref> shows the results comparion between TextLap and GPT-4. Given the high failure rates of GPT-4 and TextLap-Float, we also present Max-IoU (suc), which represents the average MaxIoU across all successful generations. In particular, TextLap-CSS achieves the lowest failure rate and significantly outperforms TextDiffuser2 in Max-IoU. However, it fails to outperform the GPT-4 baselines. This limitation is likely attributed to insufficient training data, stemming from the limited number of visual-text samples in the InstLap-train dataset and the inconsistent distribution of layouts between InstLap-train and InstLap-Bench.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4">Automatic Graphic Designs</head><p>We conducted an experiment on content-aware graphic designs using the Crello dataset, where canvas sizes varies for each design. For CSS format layouts with integer coordinates, we include the canvas size in the prompt. We also normalize box coordinates to decimal values between 0 and 1 ("Float") and write layouts as JSON to leverage the model's code understanding. Figure <ref type="figure">4</ref> illustrates a generated example using CSS forat, where TextLap provides coordinates for each element given a text instruction and a list of elements.</p><p>background { left: 0px; top: 0px; width: 419px; height: 298px; } imageElement { left: 35px; top: 43px; width: 190px; height: 241px; } CSS style Layout Rendered Image Background 1. imageElement is located in the middle left of the canvas. 2. textElement1, and textElement2 is to the right of imageElement.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>CSS style prompt</head><p>/* A text that says: "Arnold," */ textElement2 {width: 88px; height: 19px} Spatial condition /* A text says: "I heard you were */ /* feeling a little blue! Get well soon!" */ textElement1 { width: 117px; height: 75px} /* A cartoon person blowing his nose into*/ /* a tissue. */ imageElement { width: 190px; height: 241px} textElement1 { left: 253px; top: 133px; width: 117px; height: 75px; } textElement2 { left: 268px; top: 94px; width: 88px; height: 19px; } Please arrange the layout of elements on a canvas with a width of 419 and a height of 298. /* The background is a light blue square with */ /* a white rectangle. */ background { width: 419px, height: 298px} MaxIoU&#8593; Precision&#8593; Recall&#8593; F-score&#8593; GPT-4 (CSS) 0.190 0.874 0.257 0.364 GPT-4 (rCSS) 0.209 0.934, 0.296, 0.406 GPT-4 (Float) 0.457 0.980 0.980 0.980 GPT-4 (rFloat) 0.440 0.984 0.984 0.984 TextLap-CSS 0.407 0.998 0.986 0.990 TextLap-Float 0.535 1.000 1.000 1.000</p><p>Table 5: Results on automatic graphic design on Crello.</p><p>We compare TextLap with four GPT-4 baselines that generate CSS and JSON-style layouts with both integer and float coordinates. SBERT retrieves demonstration layouts for the GPT-4 (rCSS, rFloat) baselines. Table <ref type="table">5</ref>, shows TextLap significantly outperforms its GPT-4 counterpart, demonstrating the effectiveness of instruction fine-tuning. Additionally, methods using float coordinates outperform those using integer coordinates, likely due to the complexity of adapting to different canvas sizes.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.5">Impact of Layout on Text-to-Image Generation</head><p>We evaluated the effect of layout guidance on textto-image generation using the COCO dataset. The experiment compared three models: Stable Diffusion 1.5, which serves as the layout-free baseline; TextLap + InstanceDiffusion <ref type="bibr">(Wang et al., 2024)</ref>, 7 and images are rendered by ARTIST <ref type="bibr">(Zhang et al., 2024a)</ref> and InstanceDiffusion <ref type="bibr">(Wang et al., 2024)</ref> respectively.</p><p>Method Image FID GPT-Preference SD 1.5 58.09 -InstDiff (TextLap) 58.92 72% InstDiff (True Layout) 58.39 76%</p><p>Table 6: Text-to-image generation results on the COCO dataset. Layout-guided models are compared to the layout-free baseline, Stable Diffusion (SD) 1.5.    8 great generalizability comes from a carefully designed dataset building strategy as described in Section 3. Figure <ref type="figure">5</ref> shows examples of visual-text layout generation, and more examples can be found in Appendix F.</p><p>Interactive Layout Design Figure <ref type="figure">6</ref> and <ref type="figure">E</ref>.1 shows an example of an interactive layout design, where the user can comment on existing layout designs, and TextLap can generate a new layout, fulfilling user requests. This is another emergent ability, as TextLap is never fine-tuned on conversational data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">Conclusion</head><p>This study addresses the text-to-layout task by creating an instructional dataset built upon available resources with the assistance of GPT-4v to finetune a large language model for layout planning. The fine-tuned model, TextLap, outperforms GPT-4 in object layout planning and can generate layouts with both text and object when trained on textonly and object-only layouts. TextLap provides a framework to addressing real-world graphic design challenges by building instruction-following layout datasets. It is desired if the image rendering models can be finetuned or jointly trained with InstLap. However, it is beyond the scope of this paper as InstLap aims to unveiling the graphic design ability of large language models.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">Limitations</head><p>The limitations of the paper are caused by the design of the dataset and the model architecture. In-stLap is a dataset created with complex heuristics and the help of large language models. There is still a quality gap between InstLap and high-quality human annotations. However, InstLap is much cheaper to build and can serve as a pioneer dataset to quickly verify whether language models can perform automatic graphic designs. TextLap uses the standard LLM architecture, and a special design with 2D spatial embedding layers should provide better performance. It is very expensive to pretrain such a model and is beyond the scope of this paper.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="8">Ethics Statement</head><p>Our paper introduces a new instruction tuning dataset, which acknowledges the potential ethical implications inherent in using large language models for such applications.We have taken comprehensive steps to ensure that our research adheres to the highest ethical standards, particularly with respect to data privacy and responsible use of AI. This ethics statement reflects our dedication to conducting responsible research and our commitment to advancing the field of AI in a manner that respects individual privacy rights and promotes the ethical use of technology.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C Additional Quantitative Results</head><p>Table <ref type="table">C</ref>.2 presents the results that combine open-set and close-set prompts, where TextLap models exhibit similar performance. This consolidation of results further underscores the consistent performance of TextLap models, showing that smaller LLMs customized for specific tasks can beat huge general LLMs, such as GPT4V.</p><p>Testing Methods Captions with target objects FID &#8595; MaxIoU &#8593; Fail % &#8595; Precision &#8593; Recall &#8593; F-score &#8593; GPT-4 382.0 0.292 0.956 0.989 0.989 0.989 GPT-4(r) 26.40 0.452 1.116 0.979 0.980 0.979 GPT-4(rCSS) 37.45 0.459 1.116 0.969 0.971 0.969 TextLap-S128 18.64 0.454 2.151 0.972 0.977 0.974 TextLap-D128 13.54 0.475 0.398 0.995 0.979 0.983 TextLap-D1024 14.43 0.456 0.398 0.993 0.966 0.973 TextLap-F 15.09 0.475 0.159 0.996 0.997 0.996 TextLap-CSS 19.32 0.458 0.000 0.998 1.000 0.998 Captions only GPT-4 504.0 0.005 98.566 0.012 0.012 0.012 GPT-4(r) 30.01 0.417 4.382 0.838 0.900 0.856 GPT-4(rCSS) 39.17 0.427 1.594 0.842 0.925 0.867 TextLap-S128 20.67 0.452 1.116 0.977 0.980 0.976 TextLap-D128 14.77 0.267 25.976 0.538 0.540 0.537 TextLap-D1024 16.27 0.347 9.163 0.718 0.719 0.716 TextLap-F 14.36 0.424 3.426 0.878 0.882 0.877 TextLap-CSS 18.69 0.440 0.080 0.983 0.981 0.979 Table C.1: Comparative results of close-set layout generation with 80-class COCO labels Testing Methods Caption with target objects FID &#8595; MaxIoU &#8593; Fail % &#8595; Precision &#8593; Recall &#8593; F-score &#8593; GPT-4 334.3 0.309 1.133 0.985 0.984 0.984 GPT-4(r) 27.99 0.438 1.482 0.977 0.978 0.977 GPT-4(rCSS) 32.54 0.458 0.828 0.970 0.972 0.971 TextLap-D128 8.49 0.479 0.697 0.991 0.973 0.978 TextLap-D1024 10.52 0.461 0.392 0.983 0.953 0.962 TextLap-F 9.96 0.466 0.305 0.969 0.970 0.969 TextLap-CSS 12.53 0.461 0.000 0.997 0.999 0.997 Caption only GPT-4 394.1 0.005 96.992 0.011 0.013 0.011 GPT-4(r) 43.07 0.298 6.495 0.589 0.664 0.611 GPT-4(rCSS) 53.38 0.276 1.962 0.543 0.610 0.563 TextLap-D128 8.46 0.292 22.537 0.587 0.587 0.585 TextLap-D1024 10.75 0.348 9.677 0.718 0.717 0.715 TextLap-F 9.74 0.372 3.357 0.768 0.771 0.767 TextLap-CSS 12.46 0.443 0.131 0.974 0.972 0.970 Table C.2: Comparative results on layout generation with both close-set and open-set labels.</p><p>13</p></div></body>
		</text>
</TEI>
